[ceph-users] Re: Cephfs scalability question
Hi, Is it advisable to limit the sizes of data pools or metadata pools of a cephfs filesystem for performance or other reasons? I assume you don't mean quotas for pools, right? The pool size is limited by the number and size of the OSDs, of course. I can't really say what's advisable or not, it really depends on the actual requirements. Ceph scales pretty well. Performance can be significantly improved if you use pinning with multiple active MDS servers. We are migrating to cephfs and I estimate that we will eventually end up with 10-15PB of data and ~1.5TB of metadata. Should I divide the data among multiple data pools? Perhaps even create multiple cephfs filesystems? This also depends on your actual requirements. For example, if you have lots of "cold" data you could archive it on erasure-coded pools on HDDs while "hot" data is on SSDs and maybe replicated. Multiple filesystems is an option, but if possible you should stick to one filesystem with pinning, here's an excerpt from the docs [1]: Multiple file systems do not share pools. This is particularly important for snapshots but also because no measures are in place to prevent duplicate inodes. The Ceph commands prevent this dangerous configuration. Each file system has its own set of MDS ranks. Consequently, each new file system requires more MDS daemons to operate and increases operational costs. This can be useful for increasing metadata throughput by application or user base but also adds cost to the creation of a file system. Generally, a single file system with subtree pinning is a better choice for isolating load between applications. Regards, Eugen [1] https://docs.ceph.com/en/latest/cephfs/multifs/ Zitat von Vladimir Brik : Hello Is it advisable to limit the sizes of data pools or metadata pools of a cephfs filesystem for performance or other reasons? We are migrating to cephfs and I estimate that we will eventually end up with 10-15PB of data and ~1.5TB of metadata. Should I divide the data among multiple data pools? Perhaps even create multiple cephfs filesystems? Thanks, Vlad ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Reinstalling OSD node managed by cephadm
Dear all,
I now attempted this and my host is back in the cluster but the `ceph
cephadm osd activate` does not work.
# ceph cephadm osd activate HOST
Created no osd(s) on host HOST; already created?
Using --verbose is not too helpful either:
bestcmds_sorted:
[{'flags': 8,
'help': 'Start OSD containers for existing OSDs',
'module': 'mgr',
'perm': 'rw',
'sig': [argdesc(, req=True,
name=prefix, n=1, numseen=0, prefix=cephadm),
argdesc(, req=True,
name=prefix, n=1, numseen=0, prefix=osd),
argdesc(, req=True,
name=prefix, n=1, numseen=0, prefix=activate),
argdesc(, req=True, name=host,
n=N, numseen=0)]}]
Submitting command: {'prefix': 'cephadm osd activate', 'host': ['HOST'],
'target': ('mon-mgr', '')}
submit {"prefix": "cephadm osd activate", "host": ["HOST"], "target":
["mon-mgr", ""]} to mon-mgr
How can I find out what goes wrong here?
Thanks!
Manuel
On Wed, Feb 2, 2022 at 12:43 PM Robert Sander
wrote:
> On 02.02.22 12:15, Manuel Holtgrewe wrote:
> >
> > Would this also work when renaming hosts at the same time?
> >
> > - remove host from ceph orch
> > - reinstall host with different name/IP
> > - add back host into ceph orch
> > - use ceph osd activate
> >
> > as above?
>
> That could also work as long as the OSDs are still in the CRUSH map.
> Keep in mind that the last command is
>
>ceph cephadm osd activate $HOSTNAME
>
> Regards
> --
> Robert Sander
> Heinlein Consulting GmbH
> Schwedter Str. 8/9b, 10119 Berlin
>
> https://www.heinlein-support.de
>
> Tel: 030 / 405051-43
> Fax: 030 / 405051-19
>
> Amtsgericht Berlin-Charlottenburg - HRB 220009 B
> Geschäftsführer: Peer Heinlein - Sitz: Berlin
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: v17.2.0 Quincy released
excellent work everyone! Regarding this: "Quincy does not support LevelDB. Please migrate your OSDs and monitors to RocksDB before upgrading to Quincy." Is there a convenient way to determine this for cephadm and non-cephadm setups? What happens if LevelDB is still active? does it cause an immediate failure to upgrade? thanks. ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: v17.2.0 Quincy released
On Wed, Apr 20, 2022 at 6:21 AM Harry G. Coin wrote: > > Great news! Any notion when the many pending bug fixes will show up in > Pacific? It's been a while. Hi Harry, The 16.2.8 release is planned within the next week or two. Thanks, Ilya ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] cephadm filter OSDs
hallo everybody, i have the following hardware which consist of 3 Nodes with the following specs: * 8 HDDs 8TB * 1 SSD 900G * 2 NVME 260G i planned to use the HDDs for the OSDs and the other devices for bluestorage(db) according to the documentation 2% of storage is needed for bluestorage as i am not going to use RGW so i need for each OSD around 160GB of db storage. The ideal setup would be 4 OSDs on the SSD and 2 OSDs on each NVME. But for some reasons using the default config: ``` service_type: osd service_id: osd_spec_a placement: host_pattern: "*" spec: data_devices: rotational: 1 db_devices: rotational: 0 ``` cephadm is assigning 6 OSDs on the NVMEs and 2 on the SSDs. i tries the following config: ``` service_type: osd service_id: osd_spec_a placement: host_pattern: "*" spec: data_devices: paths: - /dev/sdc - /dev/sdd - /dev/sde - /dev/sdf db_devices: model : Micron_ --- service_type: osd service_id: osd_spec_b placement: host_pattern: "*" spec: data_devices: paths: - /dev/sdg - /dev/sdh - /dev/sdi - /dev/sdj db_devices: model : KXG60ZNV256G ``` but `ceph orch apply -i osd.yaml --dry-run` is taking forever and not generating any data. Does anybody have an idea how to setup the filter the right way ? thanks a lot, Ali ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Reinstalling OSD node managed by cephadm
Hi,
have you checked /var/log/ceph/cephadm.log for any hints?
ceph-volume.log may also provide some information
(/var/log/ceph//ceph-volume.log) what might be going on.
Zitat von Manuel Holtgrewe :
Dear all,
I now attempted this and my host is back in the cluster but the `ceph
cephadm osd activate` does not work.
# ceph cephadm osd activate HOST
Created no osd(s) on host HOST; already created?
Using --verbose is not too helpful either:
bestcmds_sorted:
[{'flags': 8,
'help': 'Start OSD containers for existing OSDs',
'module': 'mgr',
'perm': 'rw',
'sig': [argdesc(, req=True,
name=prefix, n=1, numseen=0, prefix=cephadm),
argdesc(, req=True,
name=prefix, n=1, numseen=0, prefix=osd),
argdesc(, req=True,
name=prefix, n=1, numseen=0, prefix=activate),
argdesc(, req=True, name=host,
n=N, numseen=0)]}]
Submitting command: {'prefix': 'cephadm osd activate', 'host': ['HOST'],
'target': ('mon-mgr', '')}
submit {"prefix": "cephadm osd activate", "host": ["HOST"], "target":
["mon-mgr", ""]} to mon-mgr
How can I find out what goes wrong here?
Thanks!
Manuel
On Wed, Feb 2, 2022 at 12:43 PM Robert Sander
wrote:
On 02.02.22 12:15, Manuel Holtgrewe wrote:
>
> Would this also work when renaming hosts at the same time?
>
> - remove host from ceph orch
> - reinstall host with different name/IP
> - add back host into ceph orch
> - use ceph osd activate
>
> as above?
That could also work as long as the OSDs are still in the CRUSH map.
Keep in mind that the last command is
ceph cephadm osd activate $HOSTNAME
Regards
--
Robert Sander
Heinlein Consulting GmbH
Schwedter Str. 8/9b, 10119 Berlin
https://www.heinlein-support.de
Tel: 030 / 405051-43
Fax: 030 / 405051-19
Amtsgericht Berlin-Charlottenburg - HRB 220009 B
Geschäftsführer: Peer Heinlein - Sitz: Berlin
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] EC pool OSDs getting erroneously "full" (15.2.15)
Hi fellow ceph users and developers, we've got into quite strange situation I'm not sure is not a ceph bug.. we have 4 node CEPH cluster with multiple pools. one of them is SATA EC 2+2 pool containting 4x3 10TB drives (one of tham is actually 12TB) one day, after planned downtime of fourth node, we got into strange state where there seemed to be large amount of degraded PGs to recover (we had noout set for the duration of downtime though) the weird thing was, that OSDs of that node seemed to be almost full (ie 80%) while there were almost no PGs on them according to osd df tree leading to backfilltoofull.. after some experimenting, I dropped those and recreated them, but during the recovery, we got into the same state: -31 120.0 - 112 TiB 81 TiB 80 TiB 36 GiB 456 GiB 31 TiB 72.58 1.06- root sata-archive -32 30.0 - 29 TiB 20 TiB 20 TiB 10 GiB 133 GiB 9.5 TiB 67.48 0.99- host v1a-sata-archive 5hdd 10.0 1.0 9.2 TiB 6.2 TiB 6.1 TiB 3.5 GiB 47 GiB 3.0 TiB 67.78 0.99 171 up osd.5 10hdd 10.0 1.0 9.2 TiB 6.2 TiB 6.2 TiB 3.6 GiB 48 GiB 2.9 TiB 68.06 1.00 171 up osd.10 13hdd 10.0 1.0 11 TiB 7.3 TiB 7.3 TiB 3.2 GiB 38 GiB 3.6 TiB 66.73 0.98 170 up osd.13 -33 30.0 - 27 TiB 19 TiB 18 TiB 11 GiB 139 GiB 9.0 TiB 67.39 0.99- host v1b-sata-archive 19hdd 10.0 1.0 9.2 TiB 6.1 TiB 6.1 TiB 3.5 GiB 46 GiB 3.0 TiB 67.11 0.98 171 up osd.19 28hdd 10.0 1.0 9.2 TiB 6.1 TiB 6.0 TiB 3.5 GiB 46 GiB 3.1 TiB 66.44 0.97 170 up osd.28 29hdd 10.0 1.0 9.2 TiB 6.3 TiB 6.2 TiB 3.6 GiB 48 GiB 2.9 TiB 68.61 1.00 171 up osd.29 -34 30.0 - 27 TiB 19 TiB 19 TiB 11 GiB 143 GiB 8.6 TiB 68.65 1.00- host v1c-sata-archive 30hdd 10.0 1.0 9.2 TiB 6.3 TiB 6.2 TiB 3.8 GiB 48 GiB 2.8 TiB 68.91 1.01 171 up osd.30 31hdd 10.0 1.0 9.1 TiB 6.3 TiB 6.3 TiB 3.6 GiB 48 GiB 2.8 TiB 69.20 1.01 171 up osd.31 52hdd 10.0 1.0 9.1 TiB 6.2 TiB 6.1 TiB 3.4 GiB 46 GiB 2.9 TiB 67.84 0.99 170 up osd.52 -35 30.0 - 27 TiB 24 TiB 24 TiB 4.0 GiB 41 GiB 3.5 TiB 87.13 1.27- host v1d-sata-archive 53hdd 10.0 1.0 9.2 TiB 8.1 TiB 8.0 TiB 1.3 GiB 14 GiB 1.0 TiB 88.54 1.29 81 up osd.53 54hdd 10.0 1.0 9.2 TiB 8.3 TiB 8.2 TiB 1.4 GiB 14 GiB 897 GiB 90.44 1.32 79 up osd.54 55hdd 10.0 1.0 9.1 TiB 7.5 TiB 7.5 TiB 1.3 GiB 13 GiB 1.6 TiB 82.39 1.21 62 up osd.55 the count of pgs on osd 53..55 is less then 1/2 of other OSDs but they are almost full. according to weights, this should not happen.. any idea on why could this be happening or what to check? thanks a lot in advance for hints.. with best regards nikola ciprich -- - Ing. Nikola CIPRICH LinuxBox.cz, s.r.o. 28.rijna 168, 709 00 Ostrava tel.: +420 591 166 214 fax:+420 596 621 273 mobil: +420 777 093 799 www.linuxbox.cz mobil servis: +420 737 238 656 email servis: ser...@linuxbox.cz - ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Reinstalling OSD node managed by cephadm
Dear Eugen, thanks for the hint. The output is pasted below. I can't gather any useful information from that. I also followed the instructions from https://docs.ceph.com/en/latest/cephadm/operations/#watching-cephadm-log-messages ``` ceph config set mgr mgr/cephadm/log_to_cluster_level debug ceph -W cephadm --watch-debug ``` and the relevant log output is here: - https://privatepastebin.com/?f95c66924a7ddda9#ADEposX5DCo5fb5wGv42czrxaHscnwoHB7igc3eNQMwc Cheers, Manuel [2022-04-20 10:38:01,909][ceph_volume.main][INFO ] Running command: ceph-volume lvm list --format json [2022-04-20 10:38:01,911][ceph_volume.process][INFO ] Running command: /usr/sbin/lvs --noheadings --readonly --separator=";" -a --units=b --nosuffix -S -o lv_tags,lv_path,lv_name,vg_name,lv_uuid,lv_size [2022-04-20 10:38:01,975][ceph_volume.process][INFO ] stdout ceph.block_device=/dev/ceph-2cbf3973-13a3-444a-b335-a0262cff6074/osd-block-e2ebb627-28aa-45a3-9261-d7c27bc08448,ceph.block_uuid=AbCFEt-JrjE-bfnc-HiHu-XlyH-G72j-0jwec7,ceph.cephx_lockbox_secret=,ceph.cluster_fsid=d221bc3c-8ff4-11ec-b4ba-b02628267680,ceph.cluster_name=ceph,ceph.crush_device_class=None,ceph.encrypted=0,ceph.osd_fsid=e2ebb627-28aa-45a3-9261-d7c27bc08448,ceph.osd_id=12,ceph.osdspec_affinity=all,ceph.type=block,ceph.vdo=0";"/dev/ceph-2cbf3973-13a3-444a-b335-a0262cff6074/osd-block-e2ebb627-28aa-45a3-9261-d7c27bc08448";"osd-block-e2ebb627-28aa-45a3-9261-d7c27bc08448";"ceph-2cbf3973-13a3-444a-b335-a0262cff6074";"AbCFEt-JrjE-bfnc-HiHu-XlyH-G72j-0jwec7";"1920378863616 [2022-04-20 10:38:01,975][ceph_volume.process][INFO ] stdout ceph.block_device=/dev/ceph-6f69c24e-2930-48ff-a18f-278470e558e1/osd-block-3f3d61f8-6964-4922-98cb-6620aff5cb6f,ceph.block_uuid=MS5ExP-ROK9-nVRG-l6wA-eHLL-Wjic-lMsYPM,ceph.cephx_lockbox_secret=,ceph.cluster_fsid=d221bc3c-8ff4-11ec-b4ba-b02628267680,ceph.cluster_name=ceph,ceph.crush_device_class=None,ceph.encrypted=0,ceph.osd_fsid=3f3d61f8-6964-4922-98cb-6620aff5cb6f,ceph.osd_id=25,ceph.osdspec_affinity=all,ceph.type=block,ceph.vdo=0";"/dev/ceph-6f69c24e-2930-48ff-a18f-278470e558e1/osd-block-3f3d61f8-6964-4922-98cb-6620aff5cb6f";"osd-block-3f3d61f8-6964-4922-98cb-6620aff5cb6f";"ceph-6f69c24e-2930-48ff-a18f-278470e558e1";"MS5ExP-ROK9-nVRG-l6wA-eHLL-Wjic-lMsYPM";"1920378863616 [2022-04-20 10:38:01,975][ceph_volume.process][INFO ] stdout ceph.block_device=/dev/ceph-8d9bd6eb-ce97-4940-8fed-f57f7bed3f5a/osd-block-8d0b1bad-069a-4acf-b13b-982fab58f285,ceph.block_uuid=rrJbzA-JSuc-GrSf-KsXd-VWi6-ffTW-LS5PB3,ceph.cephx_lockbox_secret=,ceph.cluster_fsid=d221bc3c-8ff4-11ec-b4ba-b02628267680,ceph.cluster_name=ceph,ceph.crush_device_class=None,ceph.encrypted=0,ceph.osd_fsid=8d0b1bad-069a-4acf-b13b-982fab58f285,ceph.osd_id=0,ceph.osdspec_affinity=all,ceph.type=block,ceph.vdo=0";"/dev/ceph-8d9bd6eb-ce97-4940-8fed-f57f7bed3f5a/osd-block-8d0b1bad-069a-4acf-b13b-982fab58f285";"osd-block-8d0b1bad-069a-4acf-b13b-982fab58f285";"ceph-8d9bd6eb-ce97-4940-8fed-f57f7bed3f5a";"rrJbzA-JSuc-GrSf-KsXd-VWi6-ffTW-LS5PB3";"1920378863616 [2022-04-20 10:38:01,975][ceph_volume.process][INFO ] stdout ceph.block_device=/dev/ceph-a8c3f39b-ec8d-4dd5-a85a-13a5773b99fa/osd-block-ff82d0d0-6d55-49c2-85cc-bb8a0a74ae89,ceph.block_uuid=CB1rmb-Onhq-Vdrh-9Ide-XpA0-IBg3-TFrSNX,ceph.cephx_lockbox_secret=,ceph.cluster_fsid=d221bc3c-8ff4-11ec-b4ba-b02628267680,ceph.cluster_name=ceph,ceph.crush_device_class=None,ceph.encrypted=0,ceph.osd_fsid=ff82d0d0-6d55-49c2-85cc-bb8a0a74ae89,ceph.osd_id=8,ceph.osdspec_affinity=all,ceph.type=block,ceph.vdo=0";"/dev/ceph-a8c3f39b-ec8d-4dd5-a85a-13a5773b99fa/osd-block-ff82d0d0-6d55-49c2-85cc-bb8a0a74ae89";"osd-block-ff82d0d0-6d55-49c2-85cc-bb8a0a74ae89";"ceph-a8c3f39b-ec8d-4dd5-a85a-13a5773b99fa";"CB1rmb-Onhq-Vdrh-9Ide-XpA0-IBg3-TFrSNX";"1920378863616 [2022-04-20 10:38:01,975][ceph_volume.process][INFO ] stdout ceph.block_device=/dev/ceph-bf94d5a5-eb04-42e3-867a-dee2886daa62/osd-block-313160a8-594d-4384-9640-68c4d8c1b6da,ceph.block_uuid=CgvBFd-wk3r-7a0y-XPiH-WoKS-VGmX-yntCse,ceph.cephx_lockbox_secret=,ceph.cluster_fsid=d221bc3c-8ff4-11ec-b4ba-b02628267680,ceph.cluster_name=ceph,ceph.crush_device_class=None,ceph.encrypted=0,ceph.osd_fsid=313160a8-594d-4384-9640-68c4d8c1b6da,ceph.osd_id=4,ceph.osdspec_affinity=all,ceph.type=block,ceph.vdo=0";"/dev/ceph-bf94d5a5-eb04-42e3-867a-dee2886daa62/osd-block-313160a8-594d-4384-9640-68c4d8c1b6da";"osd-block-313160a8-594d-4384-9640-68c4d8c1b6da";"ceph-bf94d5a5-eb04-42e3-867a-dee2886daa62";"CgvBFd-wk3r-7a0y-XPiH-WoKS-VGmX-yntCse";"1920378863616 [2022-04-20 10:38:01,975][ceph_volume.process][INFO ] stdout ceph.block_device=/dev/ceph-ee2517dc-ecb6-4c14-ab22-59d9c54f0952/osd-block-f7e67343-4fde-4e45-bc70-f44c92a178bd,ceph.block_uuid=pfWtmF-6Xlc-R2LO-kzeV-2jIw-3Ki8-gCOMwZ,ceph.cephx_lockbox_secret=,ceph.cluster_fsid=d221bc3c-8ff4-11ec-b4ba-b02628267680,ceph.cluster_name=ceph,ceph.crush_device_class=None,ceph.encrypted=0,ceph.osd_fsid=f7e67343-4fde-4e45-bc70-f44c92a178bd,ceph.osd_id=20,ceph.osdspec_affinity=all,ceph.type=bloc
[ceph-users] Re: Reinstalling OSD node managed by cephadm
Hi, and the relevant log output is here: https://privatepastebin.com/?f95c66924a7ddda9#ADEposX5DCo5fb5wGv42czrxaHscnwoHB7igc3eNQMwc This is just the output of 'ceph-volume lvm list', is that really all? I haven't had the chance to test 'ceph cephadm osd activate' myself so I can't really tell what to expect but apparently the LVs and OSDs are present, cephadm just doesn't seem to try to do anything with them. Do you see in the host's logs any attempt to start container for the OSDs? Zitat von Manuel Holtgrewe : Dear Eugen, thanks for the hint. The output is pasted below. I can't gather any useful information from that. I also followed the instructions from https://docs.ceph.com/en/latest/cephadm/operations/#watching-cephadm-log-messages ``` ceph config set mgr mgr/cephadm/log_to_cluster_level debug ceph -W cephadm --watch-debug ``` and the relevant log output is here: - https://privatepastebin.com/?f95c66924a7ddda9#ADEposX5DCo5fb5wGv42czrxaHscnwoHB7igc3eNQMwc Cheers, Manuel [2022-04-20 10:38:01,909][ceph_volume.main][INFO ] Running command: ceph-volume lvm list --format json [2022-04-20 10:38:01,911][ceph_volume.process][INFO ] Running command: /usr/sbin/lvs --noheadings --readonly --separator=";" -a --units=b --nosuffix -S -o lv_tags,lv_path,lv_name,vg_name,lv_uuid,lv_size [2022-04-20 10:38:01,975][ceph_volume.process][INFO ] stdout ceph.block_device=/dev/ceph-2cbf3973-13a3-444a-b335-a0262cff6074/osd-block-e2ebb627-28aa-45a3-9261-d7c27bc08448,ceph.block_uuid=AbCFEt-JrjE-bfnc-HiHu-XlyH-G72j-0jwec7,ceph.cephx_lockbox_secret=,ceph.cluster_fsid=d221bc3c-8ff4-11ec-b4ba-b02628267680,ceph.cluster_name=ceph,ceph.crush_device_class=None,ceph.encrypted=0,ceph.osd_fsid=e2ebb627-28aa-45a3-9261-d7c27bc08448,ceph.osd_id=12,ceph.osdspec_affinity=all,ceph.type=block,ceph.vdo=0";"/dev/ceph-2cbf3973-13a3-444a-b335-a0262cff6074/osd-block-e2ebb627-28aa-45a3-9261-d7c27bc08448";"osd-block-e2ebb627-28aa-45a3-9261-d7c27bc08448";"ceph-2cbf3973-13a3-444a-b335-a0262cff6074";"AbCFEt-JrjE-bfnc-HiHu-XlyH-G72j-0jwec7";"1920378863616 [2022-04-20 10:38:01,975][ceph_volume.process][INFO ] stdout ceph.block_device=/dev/ceph-6f69c24e-2930-48ff-a18f-278470e558e1/osd-block-3f3d61f8-6964-4922-98cb-6620aff5cb6f,ceph.block_uuid=MS5ExP-ROK9-nVRG-l6wA-eHLL-Wjic-lMsYPM,ceph.cephx_lockbox_secret=,ceph.cluster_fsid=d221bc3c-8ff4-11ec-b4ba-b02628267680,ceph.cluster_name=ceph,ceph.crush_device_class=None,ceph.encrypted=0,ceph.osd_fsid=3f3d61f8-6964-4922-98cb-6620aff5cb6f,ceph.osd_id=25,ceph.osdspec_affinity=all,ceph.type=block,ceph.vdo=0";"/dev/ceph-6f69c24e-2930-48ff-a18f-278470e558e1/osd-block-3f3d61f8-6964-4922-98cb-6620aff5cb6f";"osd-block-3f3d61f8-6964-4922-98cb-6620aff5cb6f";"ceph-6f69c24e-2930-48ff-a18f-278470e558e1";"MS5ExP-ROK9-nVRG-l6wA-eHLL-Wjic-lMsYPM";"1920378863616 [2022-04-20 10:38:01,975][ceph_volume.process][INFO ] stdout ceph.block_device=/dev/ceph-8d9bd6eb-ce97-4940-8fed-f57f7bed3f5a/osd-block-8d0b1bad-069a-4acf-b13b-982fab58f285,ceph.block_uuid=rrJbzA-JSuc-GrSf-KsXd-VWi6-ffTW-LS5PB3,ceph.cephx_lockbox_secret=,ceph.cluster_fsid=d221bc3c-8ff4-11ec-b4ba-b02628267680,ceph.cluster_name=ceph,ceph.crush_device_class=None,ceph.encrypted=0,ceph.osd_fsid=8d0b1bad-069a-4acf-b13b-982fab58f285,ceph.osd_id=0,ceph.osdspec_affinity=all,ceph.type=block,ceph.vdo=0";"/dev/ceph-8d9bd6eb-ce97-4940-8fed-f57f7bed3f5a/osd-block-8d0b1bad-069a-4acf-b13b-982fab58f285";"osd-block-8d0b1bad-069a-4acf-b13b-982fab58f285";"ceph-8d9bd6eb-ce97-4940-8fed-f57f7bed3f5a";"rrJbzA-JSuc-GrSf-KsXd-VWi6-ffTW-LS5PB3";"1920378863616 [2022-04-20 10:38:01,975][ceph_volume.process][INFO ] stdout ceph.block_device=/dev/ceph-a8c3f39b-ec8d-4dd5-a85a-13a5773b99fa/osd-block-ff82d0d0-6d55-49c2-85cc-bb8a0a74ae89,ceph.block_uuid=CB1rmb-Onhq-Vdrh-9Ide-XpA0-IBg3-TFrSNX,ceph.cephx_lockbox_secret=,ceph.cluster_fsid=d221bc3c-8ff4-11ec-b4ba-b02628267680,ceph.cluster_name=ceph,ceph.crush_device_class=None,ceph.encrypted=0,ceph.osd_fsid=ff82d0d0-6d55-49c2-85cc-bb8a0a74ae89,ceph.osd_id=8,ceph.osdspec_affinity=all,ceph.type=block,ceph.vdo=0";"/dev/ceph-a8c3f39b-ec8d-4dd5-a85a-13a5773b99fa/osd-block-ff82d0d0-6d55-49c2-85cc-bb8a0a74ae89";"osd-block-ff82d0d0-6d55-49c2-85cc-bb8a0a74ae89";"ceph-a8c3f39b-ec8d-4dd5-a85a-13a5773b99fa";"CB1rmb-Onhq-Vdrh-9Ide-XpA0-IBg3-TFrSNX";"1920378863616 [2022-04-20 10:38:01,975][ceph_volume.process][INFO ] stdout ceph.block_device=/dev/ceph-bf94d5a5-eb04-42e3-867a-dee2886daa62/osd-block-313160a8-594d-4384-9640-68c4d8c1b6da,ceph.block_uuid=CgvBFd-wk3r-7a0y-XPiH-WoKS-VGmX-yntCse,ceph.cephx_lockbox_secret=,ceph.cluster_fsid=d221bc3c-8ff4-11ec-b4ba-b02628267680,ceph.cluster_name=ceph,ceph.crush_device_class=None,ceph.encrypted=0,ceph.osd_fsid=313160a8-594d-4384-9640-68c4d8c1b6da,ceph.osd_id=4,ceph.osdspec_affinity=all,ceph.type=block,ceph.vdo=0";"/dev/ceph-bf94d5a5-eb04-42e3-867a-dee2886daa62/osd-block-313160a8-594d-4384-9640-68c4d8c1b6da";"osd-block-313160a8-594d-4384-9640-68c4d8c1b6da";"ceph-bf94d5a5-eb04-42e3-867a-dee2886daa62";"CgvBFd-
[ceph-users] Slow read write operation in ssd disk pool
Dear Team, We have two type to disk in our ceph cluster one is Magnetic disk, another type is SSD. # ceph osd crush class ls [ "hdd", "ssd" ] hdd is normally a bit slower, which is normal. initially ssd was faster read/write. Recently we are facing very slow operation on ssd. Need help to troubleshoot the issue we are using CEPH in our openstack cinder backend Regards, Munna ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: EC pool OSDs getting erroneously "full" (15.2.15)
Hi Stefan,
all daemons are 15.2.15 (I'm considering doing update to 15.2.16 today)
> What do you have set as neafull ratio? ceph osd dump |grep nearfull.
nearfull is 0.87
>
> Do you have the ceph balancer enabled? ceph balancer status
{
"active": true,
"last_optimize_duration": "0:00:00.000538",
"last_optimize_started": "Wed Apr 20 13:02:26 2022",
"mode": "crush-compat",
"optimize_result": "Some objects (0.130412) are degraded; try again later",
"plans": []
}
> What kind of maintenance was going on?
we were replacing failing memory module (according to IPMI log, all errors
were corrected though..)
>
> Are the PGs on those OSDs *way* bigger than on those of the other nodes?
> ceph pg ls-by-osd $osd-id and check for bytes (and OMAP bytes). Only
> accurate information when PGs have been recently deep-scrubbed.
sizes seem to be ~similar (each pg is between 65-75GB), if I count sum of them,
it's almost twice as big for osd.5 as for osd.53-osd.55
it hasn't been scrubbed due to ongoing recovery though.. but the OMAP
sizes shouldn't make such a difference..
>
> In this case the PG backfilltoofull warning(s) might have been correct.
> Yesterday though, I had no OSDs close to near full ratio and was getting the
> same PG backfilltoofull message ... previously seen due to this bug [1]. I
> could fix that by setting upmaps for the affacted PGs to another OSD.
warning is correct, but the usage value seems to be wrong..
what comes to my mind, there seem to be a lot of pgs waiting for snaptrims..
I'll keep it snaptrimming for some time and see if usage lowers...
>
> >
> > any idea on why could this be happening or what to check?
>
> I helps to know what kind of maintenance was going on. Sometimes Ceph PG
> mappings are not what you want. There are ways to do maintenance in a more
> controlled fashion.
the maintenance itself wasn't ceph related, it shouldn't cause any PG
movements..
one thing to note, I added SSD volume for all OSD DBs to speed up recovery, but
we've hat this problem before that, so I don't think this should be the
culprit..
BR
nik
>
> >
> > thanks a lot in advance for hints..
>
> Gr. Stefan
>
> [1]: https://tracker.ceph.com/issues/39555
>
--
-
Ing. Nikola CIPRICH
LinuxBox.cz, s.r.o.
28.rijna 168, 709 00 Ostrava
tel.: +420 591 166 214
fax:+420 596 621 273
mobil: +420 777 093 799
www.linuxbox.cz
mobil servis: +420 737 238 656
email servis: ser...@linuxbox.cz
-
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Reinstalling OSD node managed by cephadm
Hi,
thank you for your reply.
That really is all. I tried to call `cephadm ceph-volume lvm activate
--all`, see below, and this apparently crashes because of some unicode
problem... might that be the root cause?
Cheers,
Manuel
[root@dmz-host-4 rocky]# cephadm ceph-volume lvm activate --all
Inferring fsid d221bc3c-8ff4-11ec-b4ba-b02628267680
Using recent ceph image
quay.io/ceph/ceph@sha256:0d927ccbd8892180ee09894c2b2c26d07c938bf96a56eaee9b80fc9f26083ddb
Non-zero exit code 1 from /bin/docker run --rm --ipc=host
--stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume
--privileged --group-add=disk --init -e CONTAINER_IMAGE=
quay.io/ceph/ceph@sha256:0d927ccbd8892180ee09894c2b2c26d07c938bf96a56eaee9b80fc9f26083ddb
-e NODE_NAME=dmz-host-4 -e CEPH_USE_RANDOM_NONCE=1 -v
/var/run/ceph/d221bc3c-8ff4-11ec-b4ba-b02628267680:/var/run/ceph:z -v
/var/log/ceph/d221bc3c-8ff4-11ec-b4ba-b02628267680:/var/log/ceph:z -v
/var/lib/ceph/d221bc3c-8ff4-11ec-b4ba-b02628267680/crash:/var/lib/ceph/crash:z
-v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v
/run/lock/lvm:/run/lock/lvm -v /tmp/ceph-tmp1rkweygp:/etc/ceph/ceph.conf:z
quay.io/ceph/ceph@sha256:0d927ccbd8892180ee09894c2b2c26d07c938bf96a56eaee9b80fc9f26083ddb
lvm activate --all
/bin/docker: stderr --> Activating OSD ID 12 FSID
e2ebb627-28aa-45a3-9261-d7c27bc08448
/bin/docker: stderr Running command: /usr/bin/mount -t tmpfs tmpfs
/var/lib/ceph/osd/ceph-12
/bin/docker: stderr Running command: /usr/bin/chown -R ceph:ceph
/var/lib/ceph/osd/ceph-12
/bin/docker: stderr Running command: /usr/bin/ceph-bluestore-tool
--cluster=ceph prime-osd-dir --dev
/dev/ceph-2cbf3973-13a3-444a-b335-a0262cff6074/osd-block-e2ebb627-28aa-45a3-9261-d7c27bc08448
--path /var/lib/ceph/osd/ceph-12 --no-mon-config
/bin/docker: stderr Running command: /usr/bin/ln -snf
/dev/ceph-2cbf3973-13a3-444a-b335-a0262cff6074/osd-block-e2ebb627-28aa-45a3-9261-d7c27bc08448
/var/lib/ceph/osd/ceph-12/block
/bin/docker: stderr Running command: /usr/bin/chown -h ceph:ceph
/var/lib/ceph/osd/ceph-12/block
/bin/docker: stderr Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3
/bin/docker: stderr Running command: /usr/bin/chown -R ceph:ceph
/var/lib/ceph/osd/ceph-12
/bin/docker: stderr Running command: /usr/bin/systemctl enable
ceph-volume@lvm-12-e2ebb627-28aa-45a3-9261-d7c27bc08448
/bin/docker: stderr stderr: Created symlink
/etc/systemd/system/multi-user.target.wants/ceph-volume@lvm-12-e2ebb627-28aa-45a3-9261-d7c27bc08448.service
\u2192 /usr/lib/systemd/system/ceph-volume@.service.
/bin/docker: stderr --- Logging error ---
/bin/docker: stderr Traceback (most recent call last):
/bin/docker: stderr File "/usr/lib64/python3.6/logging/__init__.py", line
996, in emit
/bin/docker: stderr stream.write(msg)
/bin/docker: stderr UnicodeEncodeError: 'ascii' codec can't encode
character '\u2192' in position 186: ordinal not in range(128)
/bin/docker: stderr Call stack:
/bin/docker: stderr File "/usr/sbin/ceph-volume", line 11, in
/bin/docker: stderr load_entry_point('ceph-volume==1.0.0',
'console_scripts', 'ceph-volume')()
/bin/docker: stderr File
"/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 40, in __init__
/bin/docker: stderr self.main(self.argv)
/bin/docker: stderr File
"/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in
newfunc
/bin/docker: stderr return f(*a, **kw)
/bin/docker: stderr File
"/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 152, in main
/bin/docker: stderr terminal.dispatch(self.mapper, subcommand_args)
/bin/docker: stderr File
"/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in
dispatch
/bin/docker: stderr instance.main()
/bin/docker: stderr File
"/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line
46, in main
/bin/docker: stderr terminal.dispatch(self.mapper, self.argv)
/bin/docker: stderr File
"/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in
dispatch
/bin/docker: stderr instance.main()
/bin/docker: stderr File
"/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/activate.py",
line 373, in main
/bin/docker: stderr self.activate_all(args)
/bin/docker: stderr File
"/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in
is_root
/bin/docker: stderr return func(*a, **kw)
/bin/docker: stderr File
"/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/activate.py",
line 254, in activate_all
/bin/docker: stderr self.activate(args, osd_id=osd_id,
osd_fsid=osd_fsid)
/bin/docker: stderr File
"/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in
is_root
/bin/docker: stderr return func(*a, **kw)
/bin/docker: stderr File
"/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/activate.py",
line 299, in activate
/bin/docker: stderr activate_bluestore(lvs, args.no_systemd)
/bin/docker: stderr File
"/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/
[ceph-users] MDS upgrade to Quincy
The Quincy release notes state that "MDS upgrades no longer require all standby MDS daemons to be stoped before upgrading a file systems's sole active MDS." but the "Upgrading non-cephadm clusters" instructions still include reducing ranks to 1, upgrading, then raising it again. Does the new feature only apply once you have upgraded to Quincy, or do the MDS upgrade notes need adjusting now? (We're upgrading from Pacific). Looking forward to trying it (on a test cluster!). Thanks, Chris ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Reinstalling OSD node managed by cephadm
IIUC it's just the arrow that can't be displayed when the systemd-unit
is enabled and the symlinks in
/etc/systemd/system/multi-user.target.wants/ are created. When OSDs
are created by cephadm the flag --no-systemd is usually used, can you
try this command?
cephadm ceph-volume lvm activate --all --no-systemd
Zitat von Manuel Holtgrewe :
Hi,
thank you for your reply.
That really is all. I tried to call `cephadm ceph-volume lvm activate
--all`, see below, and this apparently crashes because of some unicode
problem... might that be the root cause?
Cheers,
Manuel
[root@dmz-host-4 rocky]# cephadm ceph-volume lvm activate --all
Inferring fsid d221bc3c-8ff4-11ec-b4ba-b02628267680
Using recent ceph image
quay.io/ceph/ceph@sha256:0d927ccbd8892180ee09894c2b2c26d07c938bf96a56eaee9b80fc9f26083ddb
Non-zero exit code 1 from /bin/docker run --rm --ipc=host
--stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume
--privileged --group-add=disk --init -e CONTAINER_IMAGE=
quay.io/ceph/ceph@sha256:0d927ccbd8892180ee09894c2b2c26d07c938bf96a56eaee9b80fc9f26083ddb
-e NODE_NAME=dmz-host-4 -e CEPH_USE_RANDOM_NONCE=1 -v
/var/run/ceph/d221bc3c-8ff4-11ec-b4ba-b02628267680:/var/run/ceph:z -v
/var/log/ceph/d221bc3c-8ff4-11ec-b4ba-b02628267680:/var/log/ceph:z -v
/var/lib/ceph/d221bc3c-8ff4-11ec-b4ba-b02628267680/crash:/var/lib/ceph/crash:z
-v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v
/run/lock/lvm:/run/lock/lvm -v /tmp/ceph-tmp1rkweygp:/etc/ceph/ceph.conf:z
quay.io/ceph/ceph@sha256:0d927ccbd8892180ee09894c2b2c26d07c938bf96a56eaee9b80fc9f26083ddb
lvm activate --all
/bin/docker: stderr --> Activating OSD ID 12 FSID
e2ebb627-28aa-45a3-9261-d7c27bc08448
/bin/docker: stderr Running command: /usr/bin/mount -t tmpfs tmpfs
/var/lib/ceph/osd/ceph-12
/bin/docker: stderr Running command: /usr/bin/chown -R ceph:ceph
/var/lib/ceph/osd/ceph-12
/bin/docker: stderr Running command: /usr/bin/ceph-bluestore-tool
--cluster=ceph prime-osd-dir --dev
/dev/ceph-2cbf3973-13a3-444a-b335-a0262cff6074/osd-block-e2ebb627-28aa-45a3-9261-d7c27bc08448
--path /var/lib/ceph/osd/ceph-12 --no-mon-config
/bin/docker: stderr Running command: /usr/bin/ln -snf
/dev/ceph-2cbf3973-13a3-444a-b335-a0262cff6074/osd-block-e2ebb627-28aa-45a3-9261-d7c27bc08448
/var/lib/ceph/osd/ceph-12/block
/bin/docker: stderr Running command: /usr/bin/chown -h ceph:ceph
/var/lib/ceph/osd/ceph-12/block
/bin/docker: stderr Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3
/bin/docker: stderr Running command: /usr/bin/chown -R ceph:ceph
/var/lib/ceph/osd/ceph-12
/bin/docker: stderr Running command: /usr/bin/systemctl enable
ceph-volume@lvm-12-e2ebb627-28aa-45a3-9261-d7c27bc08448
/bin/docker: stderr stderr: Created symlink
/etc/systemd/system/multi-user.target.wants/ceph-volume@lvm-12-e2ebb627-28aa-45a3-9261-d7c27bc08448.service
\u2192 /usr/lib/systemd/system/ceph-volume@.service.
/bin/docker: stderr --- Logging error ---
/bin/docker: stderr Traceback (most recent call last):
/bin/docker: stderr File "/usr/lib64/python3.6/logging/__init__.py", line
996, in emit
/bin/docker: stderr stream.write(msg)
/bin/docker: stderr UnicodeEncodeError: 'ascii' codec can't encode
character '\u2192' in position 186: ordinal not in range(128)
/bin/docker: stderr Call stack:
/bin/docker: stderr File "/usr/sbin/ceph-volume", line 11, in
/bin/docker: stderr load_entry_point('ceph-volume==1.0.0',
'console_scripts', 'ceph-volume')()
/bin/docker: stderr File
"/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 40, in __init__
/bin/docker: stderr self.main(self.argv)
/bin/docker: stderr File
"/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in
newfunc
/bin/docker: stderr return f(*a, **kw)
/bin/docker: stderr File
"/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 152, in main
/bin/docker: stderr terminal.dispatch(self.mapper, subcommand_args)
/bin/docker: stderr File
"/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in
dispatch
/bin/docker: stderr instance.main()
/bin/docker: stderr File
"/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line
46, in main
/bin/docker: stderr terminal.dispatch(self.mapper, self.argv)
/bin/docker: stderr File
"/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in
dispatch
/bin/docker: stderr instance.main()
/bin/docker: stderr File
"/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/activate.py",
line 373, in main
/bin/docker: stderr self.activate_all(args)
/bin/docker: stderr File
"/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in
is_root
/bin/docker: stderr return func(*a, **kw)
/bin/docker: stderr File
"/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/activate.py",
line 254, in activate_all
/bin/docker: stderr self.activate(args, osd_id=osd_id,
osd_fsid=osd_fsid)
/bin/docker: stderr File
"/usr/lib/python3.6/site-packages/ceph_volume/d
[ceph-users] Re: Reinstalling OSD node managed by cephadm
Hm, not much more luck: # cephadm --verbose ceph-volume lvm activate --all --no-systemd cephadm ['--verbose', 'ceph-volume', 'lvm', 'activate', '--all', '--no-systemd'] Using default config: /etc/ceph/ceph.conf /bin/docker: 920d72f71171,79MiB / 251.5GiB /bin/docker: 72570e59f28e,77.43MiB / 251.5GiB /bin/docker: b1a0de17c02d,854.8MiB / 251.5GiB /bin/docker: cce740906d36,1.158GiB / 251.5GiB /bin/docker: 4fba9fb86122,209.3MiB / 251.5GiB /bin/docker: 9b06ce4f696b,96.83MiB / 251.5GiB /bin/docker: 35269d267dfc,91.12MiB / 251.5GiB /bin/docker: 54a2a92d68fa,278.1MiB / 251.5GiB /bin/docker: cee7af4d922d,871.8MiB / 251.5GiB /bin/docker: f821b8d5e473,627.8MiB / 251.5GiB /bin/docker: 805061ed9471,473.3MiB / 251.5GiB /bin/docker: 52e339e96d47,451.7MiB / 251.5GiB /bin/docker: 542bda96ee76,409.9MiB / 251.5GiB /bin/docker: 1d76652acb43,192MiB / 251.5GiB /bin/docker: 0934566689b5,120.9MiB / 251.5GiB /bin/docker: 17c036fbb15d,284.9MiB / 251.5GiB /bin/docker: 63d0925cfc81,1.646GiB / 251.5GiB /bin/docker: 06e96a3e5cbd,572.8MiB / 251.5GiB /bin/docker: 2d2fc47ec89a,19.59MiB / 251.5GiB /bin/docker: 26adae09c237,233.6MiB / 251.5GiB /bin/docker: 86cf7b732b13,90.63MiB / 251.5GiB /bin/docker: 92d366bd19aa,2.008MiB / 251.5GiB /bin/docker: 01e9dc38faab,208.1MiB / 251.5GiB /bin/docker: da22e3f7811c,655MiB / 251.5GiB /bin/docker: d88137996826,783.1MiB / 251.5GiB /bin/docker: a9f1e48f4a04,585.5MiB / 251.5GiB /bin/docker: 49742307a5b4,478.1MiB / 251.5GiB /bin/docker: 2bb7d623ed93,205.7MiB / 251.5GiB /bin/docker: 0b323f254217,243.5MiB / 251.5GiB /bin/docker: 8835104b0515,123.5MiB / 251.5GiB /bin/docker: 67472592f1b4,658.9MiB / 251.5GiB /bin/docker: 5aef3941d33b,474.4MiB / 251.5GiB /bin/docker: 0d571f24c0e9,100.5MiB / 251.5GiB /bin/docker: 7e706a832e4a,2.11GiB / 251.5GiB /bin/docker: 23a669b4fd7c,1.164GiB / 251.5GiB /bin/docker: 74581172b6fb,1.91MiB / 251.5GiB /bin/docker: 37e980f9f83d,2.258MiB / 251.5GiB /bin/docker: 858a9b82375d,178.5MiB / 251.5GiB /bin/docker: dc97abdfc393,16.77MiB / 251.5GiB /bin/docker: d561d2aa6053,5.379MiB / 251.5GiB /bin/docker: 804b450b057f,655.4MiB / 251.5GiB /bin/docker: 011180b66a7a,2.633MiB / 251.5GiB /bin/docker: 21b984cdc711,56.98MiB / 251.5GiB /bin/docker: c197971ee2ba,1.754MiB / 251.5GiB /bin/docker: a10112ce1fa7,1.887MiB / 251.5GiB /bin/docker: 546536f82ada,160.5MiB / 251.5GiB /bin/docker: 5822da42e73b,38.15MiB / 251.5GiB /bin/docker: 38e044b7d42f,6.645MiB / 251.5GiB Inferring fsid d221bc3c-8ff4-11ec-b4ba-b02628267680 /bin/docker: quay.io/ceph/ceph@sha256:0d927ccbd8892180ee09894c2b2c26d07c938bf96a56eaee9b80fc9f26083ddb /bin/docker: quay.io/ceph/ceph@ Using recent ceph image quay.io/ceph/ceph@sha256:0d927ccbd8892180ee09894c2b2c26d07c938bf96a56eaee9b80fc9f26083ddb stat: 167 167 Acquiring lock 140627186853536 on /run/cephadm/d221bc3c-8ff4-11ec-b4ba-b02628267680.lock Lock 140627186853536 acquired on /run/cephadm/d221bc3c-8ff4-11ec-b4ba-b02628267680.lock sestatus: SELinux status: disabled sestatus: SELinux status: disabled /bin/docker: --> Activating OSD ID 12 FSID e2ebb627-28aa-45a3-9261-d7c27bc08448 /bin/docker: Running command: /usr/bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-12 /bin/docker: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-12 /bin/docker: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/ceph-2cbf3973-13a3-444a-b335-a0262cff6074/osd-block-e2ebb627-28aa-45a3-9261-d7c27bc08448 --path /var/lib/ceph/osd/ceph-12 --no-mon-config /bin/docker: Running command: /usr/bin/ln -snf /dev/ceph-2cbf3973-13a3-444a-b335-a0262cff6074/osd-block-e2ebb627-28aa-45a3-9261-d7c27bc08448 /var/lib/ceph/osd/ceph-12/block /bin/docker: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-12/block /bin/docker: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 /bin/docker: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-12 /bin/docker: --> ceph-volume lvm activate successful for osd ID: 12 /bin/docker: --> Activating OSD ID 25 FSID 3f3d61f8-6964-4922-98cb-6620aff5cb6f /bin/docker: Running command: /usr/bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-25 /bin/docker: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-25 /bin/docker: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/ceph-6f69c24e-2930-48ff-a18f-278470e558e1/osd-block-3f3d61f8-6964-4922-98cb-6620aff5cb6f --path /var/lib/ceph/osd/ceph-25 --no-mon-config /bin/docker: Running command: /usr/bin/ln -snf /dev/ceph-6f69c24e-2930-48ff-a18f-278470e558e1/osd-block-3f3d61f8-6964-4922-98cb-6620aff5cb6f /var/lib/ceph/osd/ceph-25/block /bin/docker: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-25/block /bin/docker: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-6 /bin/docker: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-25 /bin/docker: --> ceph-volume lvm activate
[ceph-users] Re: EC pool OSDs getting erroneously "full" (15.2.15)
thanks for the tip on alternative balancer, I'll have a look at it however I don't think the root of the problem is in improper balancing, those 3 OSDs just should not be that full. I'll see how it gets when the snaptrims finis, usage seems to go down by 0.01%/minute now.. I'll report the results later.. > If your clients allow (understand upmaps) you might yield better results > with the balancer in upmap mode. Jonas Jelten made a nice balancer as well > [1]. > > Gr. Stefan > > [1]: https://github.com/TheJJ/ceph-balancer > -- - Ing. Nikola CIPRICH LinuxBox.cz, s.r.o. 28.rijna 168, 709 00 Ostrava tel.: +420 591 166 214 fax:+420 596 621 273 mobil: +420 777 093 799 www.linuxbox.cz mobil servis: +420 737 238 656 email servis: ser...@linuxbox.cz - ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Reinstalling OSD node managed by cephadm
Well, at least it reports that the OSDs were activated successfully: /bin/docker: --> ceph-volume lvm activate successful for osd ID: 12 /bin/docker: --> Activating OSD ID 25 FSID 3f3d61f8-6964-4922-98cb-6620aff5cb6f Now you need to get the pods up, I'm not sure if cephadm will manage that automatically or if you need to create anything yourself. Can you try to start it via 'systemctl start ceph-d221bc3c-8ff4-11ec-b4ba-b02628267680@osd.12' ? Zitat von Manuel Holtgrewe : Hm, not much more luck: # cephadm --verbose ceph-volume lvm activate --all --no-systemd cephadm ['--verbose', 'ceph-volume', 'lvm', 'activate', '--all', '--no-systemd'] Using default config: /etc/ceph/ceph.conf /bin/docker: 920d72f71171,79MiB / 251.5GiB /bin/docker: 72570e59f28e,77.43MiB / 251.5GiB /bin/docker: b1a0de17c02d,854.8MiB / 251.5GiB /bin/docker: cce740906d36,1.158GiB / 251.5GiB /bin/docker: 4fba9fb86122,209.3MiB / 251.5GiB /bin/docker: 9b06ce4f696b,96.83MiB / 251.5GiB /bin/docker: 35269d267dfc,91.12MiB / 251.5GiB /bin/docker: 54a2a92d68fa,278.1MiB / 251.5GiB /bin/docker: cee7af4d922d,871.8MiB / 251.5GiB /bin/docker: f821b8d5e473,627.8MiB / 251.5GiB /bin/docker: 805061ed9471,473.3MiB / 251.5GiB /bin/docker: 52e339e96d47,451.7MiB / 251.5GiB /bin/docker: 542bda96ee76,409.9MiB / 251.5GiB /bin/docker: 1d76652acb43,192MiB / 251.5GiB /bin/docker: 0934566689b5,120.9MiB / 251.5GiB /bin/docker: 17c036fbb15d,284.9MiB / 251.5GiB /bin/docker: 63d0925cfc81,1.646GiB / 251.5GiB /bin/docker: 06e96a3e5cbd,572.8MiB / 251.5GiB /bin/docker: 2d2fc47ec89a,19.59MiB / 251.5GiB /bin/docker: 26adae09c237,233.6MiB / 251.5GiB /bin/docker: 86cf7b732b13,90.63MiB / 251.5GiB /bin/docker: 92d366bd19aa,2.008MiB / 251.5GiB /bin/docker: 01e9dc38faab,208.1MiB / 251.5GiB /bin/docker: da22e3f7811c,655MiB / 251.5GiB /bin/docker: d88137996826,783.1MiB / 251.5GiB /bin/docker: a9f1e48f4a04,585.5MiB / 251.5GiB /bin/docker: 49742307a5b4,478.1MiB / 251.5GiB /bin/docker: 2bb7d623ed93,205.7MiB / 251.5GiB /bin/docker: 0b323f254217,243.5MiB / 251.5GiB /bin/docker: 8835104b0515,123.5MiB / 251.5GiB /bin/docker: 67472592f1b4,658.9MiB / 251.5GiB /bin/docker: 5aef3941d33b,474.4MiB / 251.5GiB /bin/docker: 0d571f24c0e9,100.5MiB / 251.5GiB /bin/docker: 7e706a832e4a,2.11GiB / 251.5GiB /bin/docker: 23a669b4fd7c,1.164GiB / 251.5GiB /bin/docker: 74581172b6fb,1.91MiB / 251.5GiB /bin/docker: 37e980f9f83d,2.258MiB / 251.5GiB /bin/docker: 858a9b82375d,178.5MiB / 251.5GiB /bin/docker: dc97abdfc393,16.77MiB / 251.5GiB /bin/docker: d561d2aa6053,5.379MiB / 251.5GiB /bin/docker: 804b450b057f,655.4MiB / 251.5GiB /bin/docker: 011180b66a7a,2.633MiB / 251.5GiB /bin/docker: 21b984cdc711,56.98MiB / 251.5GiB /bin/docker: c197971ee2ba,1.754MiB / 251.5GiB /bin/docker: a10112ce1fa7,1.887MiB / 251.5GiB /bin/docker: 546536f82ada,160.5MiB / 251.5GiB /bin/docker: 5822da42e73b,38.15MiB / 251.5GiB /bin/docker: 38e044b7d42f,6.645MiB / 251.5GiB Inferring fsid d221bc3c-8ff4-11ec-b4ba-b02628267680 /bin/docker: quay.io/ceph/ceph@sha256:0d927ccbd8892180ee09894c2b2c26d07c938bf96a56eaee9b80fc9f26083ddb /bin/docker: quay.io/ceph/ceph@ Using recent ceph image quay.io/ceph/ceph@sha256:0d927ccbd8892180ee09894c2b2c26d07c938bf96a56eaee9b80fc9f26083ddb stat: 167 167 Acquiring lock 140627186853536 on /run/cephadm/d221bc3c-8ff4-11ec-b4ba-b02628267680.lock Lock 140627186853536 acquired on /run/cephadm/d221bc3c-8ff4-11ec-b4ba-b02628267680.lock sestatus: SELinux status: disabled sestatus: SELinux status: disabled /bin/docker: --> Activating OSD ID 12 FSID e2ebb627-28aa-45a3-9261-d7c27bc08448 /bin/docker: Running command: /usr/bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-12 /bin/docker: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-12 /bin/docker: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/ceph-2cbf3973-13a3-444a-b335-a0262cff6074/osd-block-e2ebb627-28aa-45a3-9261-d7c27bc08448 --path /var/lib/ceph/osd/ceph-12 --no-mon-config /bin/docker: Running command: /usr/bin/ln -snf /dev/ceph-2cbf3973-13a3-444a-b335-a0262cff6074/osd-block-e2ebb627-28aa-45a3-9261-d7c27bc08448 /var/lib/ceph/osd/ceph-12/block /bin/docker: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-12/block /bin/docker: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 /bin/docker: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-12 /bin/docker: --> ceph-volume lvm activate successful for osd ID: 12 /bin/docker: --> Activating OSD ID 25 FSID 3f3d61f8-6964-4922-98cb-6620aff5cb6f /bin/docker: Running command: /usr/bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-25 /bin/docker: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-25 /bin/docker: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/ceph-6f69c24e-2930-48ff-a18f-278470e558e1/osd-block-3f3d61f8-6964-4922-98cb-6620aff5cb6f --path /var/l
[ceph-users] Re: Reinstalling OSD node managed by cephadm
Thank you for your reply. However, the cephadm command did not create the osd.X directories in /var/lib/ceph/FSID... Subsequently the start fails which is also shown in the journalctl output: Apr 20 14:51:28 dmz-host-4 bash[2540969]: /bin/bash: /var/lib/ceph/d221bc3c-8ff4-11ec-b4ba-b02628267680/osd.12/unit.poststop: No such file or directory Best wishes, Manuel On Wed, Apr 20, 2022 at 2:42 PM Eugen Block wrote: > Well, at least it reports that the OSDs were activated successfully: > > > /bin/docker: --> ceph-volume lvm activate successful for osd ID: 12 > > /bin/docker: --> Activating OSD ID 25 FSID > > 3f3d61f8-6964-4922-98cb-6620aff5cb6f > > Now you need to get the pods up, I'm not sure if cephadm will manage > that automatically or if you need to create anything yourself. Can you > try to start it via 'systemctl start > ceph-d221bc3c-8ff4-11ec-b4ba-b02628267680@osd.12' ? > > Zitat von Manuel Holtgrewe : > > > Hm, not much more luck: > > > > # cephadm --verbose ceph-volume lvm activate --all --no-systemd > > > > > cephadm ['--verbose', 'ceph-volume', 'lvm', 'activate', '--all', > > '--no-systemd'] > > Using default config: /etc/ceph/ceph.conf > > /bin/docker: 920d72f71171,79MiB / 251.5GiB > > /bin/docker: 72570e59f28e,77.43MiB / 251.5GiB > > /bin/docker: b1a0de17c02d,854.8MiB / 251.5GiB > > /bin/docker: cce740906d36,1.158GiB / 251.5GiB > > /bin/docker: 4fba9fb86122,209.3MiB / 251.5GiB > > /bin/docker: 9b06ce4f696b,96.83MiB / 251.5GiB > > /bin/docker: 35269d267dfc,91.12MiB / 251.5GiB > > /bin/docker: 54a2a92d68fa,278.1MiB / 251.5GiB > > /bin/docker: cee7af4d922d,871.8MiB / 251.5GiB > > /bin/docker: f821b8d5e473,627.8MiB / 251.5GiB > > /bin/docker: 805061ed9471,473.3MiB / 251.5GiB > > /bin/docker: 52e339e96d47,451.7MiB / 251.5GiB > > /bin/docker: 542bda96ee76,409.9MiB / 251.5GiB > > /bin/docker: 1d76652acb43,192MiB / 251.5GiB > > /bin/docker: 0934566689b5,120.9MiB / 251.5GiB > > /bin/docker: 17c036fbb15d,284.9MiB / 251.5GiB > > /bin/docker: 63d0925cfc81,1.646GiB / 251.5GiB > > /bin/docker: 06e96a3e5cbd,572.8MiB / 251.5GiB > > /bin/docker: 2d2fc47ec89a,19.59MiB / 251.5GiB > > /bin/docker: 26adae09c237,233.6MiB / 251.5GiB > > /bin/docker: 86cf7b732b13,90.63MiB / 251.5GiB > > /bin/docker: 92d366bd19aa,2.008MiB / 251.5GiB > > /bin/docker: 01e9dc38faab,208.1MiB / 251.5GiB > > /bin/docker: da22e3f7811c,655MiB / 251.5GiB > > /bin/docker: d88137996826,783.1MiB / 251.5GiB > > /bin/docker: a9f1e48f4a04,585.5MiB / 251.5GiB > > /bin/docker: 49742307a5b4,478.1MiB / 251.5GiB > > /bin/docker: 2bb7d623ed93,205.7MiB / 251.5GiB > > /bin/docker: 0b323f254217,243.5MiB / 251.5GiB > > /bin/docker: 8835104b0515,123.5MiB / 251.5GiB > > /bin/docker: 67472592f1b4,658.9MiB / 251.5GiB > > /bin/docker: 5aef3941d33b,474.4MiB / 251.5GiB > > /bin/docker: 0d571f24c0e9,100.5MiB / 251.5GiB > > /bin/docker: 7e706a832e4a,2.11GiB / 251.5GiB > > /bin/docker: 23a669b4fd7c,1.164GiB / 251.5GiB > > /bin/docker: 74581172b6fb,1.91MiB / 251.5GiB > > /bin/docker: 37e980f9f83d,2.258MiB / 251.5GiB > > /bin/docker: 858a9b82375d,178.5MiB / 251.5GiB > > /bin/docker: dc97abdfc393,16.77MiB / 251.5GiB > > /bin/docker: d561d2aa6053,5.379MiB / 251.5GiB > > /bin/docker: 804b450b057f,655.4MiB / 251.5GiB > > /bin/docker: 011180b66a7a,2.633MiB / 251.5GiB > > /bin/docker: 21b984cdc711,56.98MiB / 251.5GiB > > /bin/docker: c197971ee2ba,1.754MiB / 251.5GiB > > /bin/docker: a10112ce1fa7,1.887MiB / 251.5GiB > > /bin/docker: 546536f82ada,160.5MiB / 251.5GiB > > /bin/docker: 5822da42e73b,38.15MiB / 251.5GiB > > /bin/docker: 38e044b7d42f,6.645MiB / 251.5GiB > > Inferring fsid d221bc3c-8ff4-11ec-b4ba-b02628267680 > > /bin/docker: > > > quay.io/ceph/ceph@sha256:0d927ccbd8892180ee09894c2b2c26d07c938bf96a56eaee9b80fc9f26083ddb > > /bin/docker: quay.io/ceph/ceph@ > > Using recent ceph image > > > quay.io/ceph/ceph@sha256:0d927ccbd8892180ee09894c2b2c26d07c938bf96a56eaee9b80fc9f26083ddb > > stat: 167 167 > > Acquiring lock 140627186853536 on > > /run/cephadm/d221bc3c-8ff4-11ec-b4ba-b02628267680.lock > > Lock 140627186853536 acquired on > > /run/cephadm/d221bc3c-8ff4-11ec-b4ba-b02628267680.lock > > sestatus: SELinux status: disabled > > sestatus: SELinux status: disabled > > /bin/docker: --> Activating OSD ID 12 FSID > > e2ebb627-28aa-45a3-9261-d7c27bc08448 > > /bin/docker: Running command: /usr/bin/mount -t tmpfs tmpfs > > /var/lib/ceph/osd/ceph-12 > > /bin/docker: Running command: /usr/bin/chown -R ceph:ceph > > /var/lib/ceph/osd/ceph-12 > > /bin/docker: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph > > prime-osd-dir --dev > > > /dev/ceph-2cbf3973-13a3-444a-b335-a0262cff6074/osd-block-e2ebb627-28aa-45a3-9261-d7c27bc08448 > > --path /var/lib/ceph/osd/ceph-12 --no-mon-config > > /bin/docker: Running command: /usr/bin/ln -snf > > > /dev/ceph-2cbf3973-13a3-444a-b335-a0262cff6074/osd-block-e2ebb627-28aa-45a3-9261-d7c27bc08448 > > /var/lib/ceph/osd/ceph-1
[ceph-users] Re: Reinstalling OSD node managed by cephadm
I see. I'm not sure if cephadm should be able to handle that and this is a bug or if you need to create those files and directories yourself. I was able to revive OSDs in a test cluster this way, but that should not be necessary. Maybe there is already an existing tracker, but if not you should probably create one. Zitat von Manuel Holtgrewe : Thank you for your reply. However, the cephadm command did not create the osd.X directories in /var/lib/ceph/FSID... Subsequently the start fails which is also shown in the journalctl output: Apr 20 14:51:28 dmz-host-4 bash[2540969]: /bin/bash: /var/lib/ceph/d221bc3c-8ff4-11ec-b4ba-b02628267680/osd.12/unit.poststop: No such file or directory Best wishes, Manuel On Wed, Apr 20, 2022 at 2:42 PM Eugen Block wrote: Well, at least it reports that the OSDs were activated successfully: > /bin/docker: --> ceph-volume lvm activate successful for osd ID: 12 > /bin/docker: --> Activating OSD ID 25 FSID > 3f3d61f8-6964-4922-98cb-6620aff5cb6f Now you need to get the pods up, I'm not sure if cephadm will manage that automatically or if you need to create anything yourself. Can you try to start it via 'systemctl start ceph-d221bc3c-8ff4-11ec-b4ba-b02628267680@osd.12' ? Zitat von Manuel Holtgrewe : > Hm, not much more luck: > > # cephadm --verbose ceph-volume lvm activate --all --no-systemd > > cephadm ['--verbose', 'ceph-volume', 'lvm', 'activate', '--all', > '--no-systemd'] > Using default config: /etc/ceph/ceph.conf > /bin/docker: 920d72f71171,79MiB / 251.5GiB > /bin/docker: 72570e59f28e,77.43MiB / 251.5GiB > /bin/docker: b1a0de17c02d,854.8MiB / 251.5GiB > /bin/docker: cce740906d36,1.158GiB / 251.5GiB > /bin/docker: 4fba9fb86122,209.3MiB / 251.5GiB > /bin/docker: 9b06ce4f696b,96.83MiB / 251.5GiB > /bin/docker: 35269d267dfc,91.12MiB / 251.5GiB > /bin/docker: 54a2a92d68fa,278.1MiB / 251.5GiB > /bin/docker: cee7af4d922d,871.8MiB / 251.5GiB > /bin/docker: f821b8d5e473,627.8MiB / 251.5GiB > /bin/docker: 805061ed9471,473.3MiB / 251.5GiB > /bin/docker: 52e339e96d47,451.7MiB / 251.5GiB > /bin/docker: 542bda96ee76,409.9MiB / 251.5GiB > /bin/docker: 1d76652acb43,192MiB / 251.5GiB > /bin/docker: 0934566689b5,120.9MiB / 251.5GiB > /bin/docker: 17c036fbb15d,284.9MiB / 251.5GiB > /bin/docker: 63d0925cfc81,1.646GiB / 251.5GiB > /bin/docker: 06e96a3e5cbd,572.8MiB / 251.5GiB > /bin/docker: 2d2fc47ec89a,19.59MiB / 251.5GiB > /bin/docker: 26adae09c237,233.6MiB / 251.5GiB > /bin/docker: 86cf7b732b13,90.63MiB / 251.5GiB > /bin/docker: 92d366bd19aa,2.008MiB / 251.5GiB > /bin/docker: 01e9dc38faab,208.1MiB / 251.5GiB > /bin/docker: da22e3f7811c,655MiB / 251.5GiB > /bin/docker: d88137996826,783.1MiB / 251.5GiB > /bin/docker: a9f1e48f4a04,585.5MiB / 251.5GiB > /bin/docker: 49742307a5b4,478.1MiB / 251.5GiB > /bin/docker: 2bb7d623ed93,205.7MiB / 251.5GiB > /bin/docker: 0b323f254217,243.5MiB / 251.5GiB > /bin/docker: 8835104b0515,123.5MiB / 251.5GiB > /bin/docker: 67472592f1b4,658.9MiB / 251.5GiB > /bin/docker: 5aef3941d33b,474.4MiB / 251.5GiB > /bin/docker: 0d571f24c0e9,100.5MiB / 251.5GiB > /bin/docker: 7e706a832e4a,2.11GiB / 251.5GiB > /bin/docker: 23a669b4fd7c,1.164GiB / 251.5GiB > /bin/docker: 74581172b6fb,1.91MiB / 251.5GiB > /bin/docker: 37e980f9f83d,2.258MiB / 251.5GiB > /bin/docker: 858a9b82375d,178.5MiB / 251.5GiB > /bin/docker: dc97abdfc393,16.77MiB / 251.5GiB > /bin/docker: d561d2aa6053,5.379MiB / 251.5GiB > /bin/docker: 804b450b057f,655.4MiB / 251.5GiB > /bin/docker: 011180b66a7a,2.633MiB / 251.5GiB > /bin/docker: 21b984cdc711,56.98MiB / 251.5GiB > /bin/docker: c197971ee2ba,1.754MiB / 251.5GiB > /bin/docker: a10112ce1fa7,1.887MiB / 251.5GiB > /bin/docker: 546536f82ada,160.5MiB / 251.5GiB > /bin/docker: 5822da42e73b,38.15MiB / 251.5GiB > /bin/docker: 38e044b7d42f,6.645MiB / 251.5GiB > Inferring fsid d221bc3c-8ff4-11ec-b4ba-b02628267680 > /bin/docker: > quay.io/ceph/ceph@sha256:0d927ccbd8892180ee09894c2b2c26d07c938bf96a56eaee9b80fc9f26083ddb > /bin/docker: quay.io/ceph/ceph@ > Using recent ceph image > quay.io/ceph/ceph@sha256:0d927ccbd8892180ee09894c2b2c26d07c938bf96a56eaee9b80fc9f26083ddb > stat: 167 167 > Acquiring lock 140627186853536 on > /run/cephadm/d221bc3c-8ff4-11ec-b4ba-b02628267680.lock > Lock 140627186853536 acquired on > /run/cephadm/d221bc3c-8ff4-11ec-b4ba-b02628267680.lock > sestatus: SELinux status: disabled > sestatus: SELinux status: disabled > /bin/docker: --> Activating OSD ID 12 FSID > e2ebb627-28aa-45a3-9261-d7c27bc08448 > /bin/docker: Running command: /usr/bin/mount -t tmpfs tmpfs > /var/lib/ceph/osd/ceph-12 > /bin/docker: Running command: /usr/bin/chown -R ceph:ceph > /var/lib/ceph/osd/ceph-12 > /bin/docker: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph > prime-osd-dir --dev > /dev/ceph-2cbf3973-13a3-444a-b335-a0262cff6074/osd-block-e2ebb627-28aa-45a3-9261-d7c27bc08448 > --path /var/lib/ceph/osd/ceph-12 --no-mon-config > /bin
[ceph-users] Re: Slow read write operation in ssd disk pool
Hi, Those are not enterprise SSD, Samsung labeled and all are 2TB in size. we have three node, 12 disk on each node Regards, Munna On Wed, Apr 20, 2022 at 5:49 PM Stefan Kooman wrote: > On 4/20/22 13:30, Md. Hejbul Tawhid MUNNA wrote: > > Dear Team, > > > > We have two type to disk in our ceph cluster one is Magnetic disk, > another > > type is SSD. > > > > # ceph osd crush class ls > > [ > > "hdd", > > "ssd" > > ] > > > > hdd is normally a bit slower, which is normal. initially ssd was faster > > read/write. Recently we are facing very slow operation on ssd. Need help > to > > troubleshoot the issue > > What Brand / model are those SSDs? Are those "enterprise" SSDs (with > wear leveling, capacitors, etc.)? > > Gr. Stefan > ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Reinstalling OSD node managed by cephadm
Hi, thank you. Done, I put the link below. Maybe someone else on this list can enlighten us ;-) https://tracker.ceph.com/issues/55395 On Wed, Apr 20, 2022 at 2:55 PM Eugen Block wrote: > I see. I'm not sure if cephadm should be able to handle that and this > is a bug or if you need to create those files and directories > yourself. I was able to revive OSDs in a test cluster this way, but > that should not be necessary. Maybe there is already an existing > tracker, but if not you should probably create one. > > Zitat von Manuel Holtgrewe : > > > Thank you for your reply. > > > > However, the cephadm command did not create the osd.X directories in > > /var/lib/ceph/FSID... Subsequently the start fails which is also shown in > > the journalctl output: > > > > Apr 20 14:51:28 dmz-host-4 bash[2540969]: /bin/bash: > > /var/lib/ceph/d221bc3c-8ff4-11ec-b4ba-b02628267680/osd.12/unit.poststop: > No > > such file or directory > > > > Best wishes, > > Manuel > > > > On Wed, Apr 20, 2022 at 2:42 PM Eugen Block wrote: > > > >> Well, at least it reports that the OSDs were activated successfully: > >> > >> > /bin/docker: --> ceph-volume lvm activate successful for osd ID: 12 > >> > /bin/docker: --> Activating OSD ID 25 FSID > >> > 3f3d61f8-6964-4922-98cb-6620aff5cb6f > >> > >> Now you need to get the pods up, I'm not sure if cephadm will manage > >> that automatically or if you need to create anything yourself. Can you > >> try to start it via 'systemctl start > >> ceph-d221bc3c-8ff4-11ec-b4ba-b02628267680@osd.12' ? > >> > >> Zitat von Manuel Holtgrewe : > >> > >> > Hm, not much more luck: > >> > > >> > # cephadm --verbose ceph-volume lvm activate --all --no-systemd > >> > > >> > > >> > cephadm ['--verbose', 'ceph-volume', 'lvm', 'activate', '--all', > >> > '--no-systemd'] > >> > Using default config: /etc/ceph/ceph.conf > >> > /bin/docker: 920d72f71171,79MiB / 251.5GiB > >> > /bin/docker: 72570e59f28e,77.43MiB / 251.5GiB > >> > /bin/docker: b1a0de17c02d,854.8MiB / 251.5GiB > >> > /bin/docker: cce740906d36,1.158GiB / 251.5GiB > >> > /bin/docker: 4fba9fb86122,209.3MiB / 251.5GiB > >> > /bin/docker: 9b06ce4f696b,96.83MiB / 251.5GiB > >> > /bin/docker: 35269d267dfc,91.12MiB / 251.5GiB > >> > /bin/docker: 54a2a92d68fa,278.1MiB / 251.5GiB > >> > /bin/docker: cee7af4d922d,871.8MiB / 251.5GiB > >> > /bin/docker: f821b8d5e473,627.8MiB / 251.5GiB > >> > /bin/docker: 805061ed9471,473.3MiB / 251.5GiB > >> > /bin/docker: 52e339e96d47,451.7MiB / 251.5GiB > >> > /bin/docker: 542bda96ee76,409.9MiB / 251.5GiB > >> > /bin/docker: 1d76652acb43,192MiB / 251.5GiB > >> > /bin/docker: 0934566689b5,120.9MiB / 251.5GiB > >> > /bin/docker: 17c036fbb15d,284.9MiB / 251.5GiB > >> > /bin/docker: 63d0925cfc81,1.646GiB / 251.5GiB > >> > /bin/docker: 06e96a3e5cbd,572.8MiB / 251.5GiB > >> > /bin/docker: 2d2fc47ec89a,19.59MiB / 251.5GiB > >> > /bin/docker: 26adae09c237,233.6MiB / 251.5GiB > >> > /bin/docker: 86cf7b732b13,90.63MiB / 251.5GiB > >> > /bin/docker: 92d366bd19aa,2.008MiB / 251.5GiB > >> > /bin/docker: 01e9dc38faab,208.1MiB / 251.5GiB > >> > /bin/docker: da22e3f7811c,655MiB / 251.5GiB > >> > /bin/docker: d88137996826,783.1MiB / 251.5GiB > >> > /bin/docker: a9f1e48f4a04,585.5MiB / 251.5GiB > >> > /bin/docker: 49742307a5b4,478.1MiB / 251.5GiB > >> > /bin/docker: 2bb7d623ed93,205.7MiB / 251.5GiB > >> > /bin/docker: 0b323f254217,243.5MiB / 251.5GiB > >> > /bin/docker: 8835104b0515,123.5MiB / 251.5GiB > >> > /bin/docker: 67472592f1b4,658.9MiB / 251.5GiB > >> > /bin/docker: 5aef3941d33b,474.4MiB / 251.5GiB > >> > /bin/docker: 0d571f24c0e9,100.5MiB / 251.5GiB > >> > /bin/docker: 7e706a832e4a,2.11GiB / 251.5GiB > >> > /bin/docker: 23a669b4fd7c,1.164GiB / 251.5GiB > >> > /bin/docker: 74581172b6fb,1.91MiB / 251.5GiB > >> > /bin/docker: 37e980f9f83d,2.258MiB / 251.5GiB > >> > /bin/docker: 858a9b82375d,178.5MiB / 251.5GiB > >> > /bin/docker: dc97abdfc393,16.77MiB / 251.5GiB > >> > /bin/docker: d561d2aa6053,5.379MiB / 251.5GiB > >> > /bin/docker: 804b450b057f,655.4MiB / 251.5GiB > >> > /bin/docker: 011180b66a7a,2.633MiB / 251.5GiB > >> > /bin/docker: 21b984cdc711,56.98MiB / 251.5GiB > >> > /bin/docker: c197971ee2ba,1.754MiB / 251.5GiB > >> > /bin/docker: a10112ce1fa7,1.887MiB / 251.5GiB > >> > /bin/docker: 546536f82ada,160.5MiB / 251.5GiB > >> > /bin/docker: 5822da42e73b,38.15MiB / 251.5GiB > >> > /bin/docker: 38e044b7d42f,6.645MiB / 251.5GiB > >> > Inferring fsid d221bc3c-8ff4-11ec-b4ba-b02628267680 > >> > /bin/docker: > >> > > >> > quay.io/ceph/ceph@sha256:0d927ccbd8892180ee09894c2b2c26d07c938bf96a56eaee9b80fc9f26083ddb > >> > /bin/docker: quay.io/ceph/ceph@ > >> > Using recent ceph image > >> > > >> > quay.io/ceph/ceph@sha256:0d927ccbd8892180ee09894c2b2c26d07c938bf96a56eaee9b80fc9f26083ddb > >> > stat: 167 167 > >> > Acquiring lock 140627186853536 on > >> > /run/cephadm/d221bc3c-8ff4-11ec-b4ba-b02628267680.lock > >> > Lock 140627186853536 acquired on > >> > /run/cephadm
[ceph-users] Re: Slow read write operation in ssd disk pool
> > Those are not enterprise SSD, Samsung labeled and all are 2TB in size. we > have three node, 12 disk on each node > I think you should use the drives labeled with the lower case 's' of samsung. ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: cephfs-top doesn't work
On 4/19/22 10:56 PM, Vladimir Brik wrote: Yeah, this must be the bug. I have about 180 clients Yeah, a new bug. Maybe too many clients and couldn't show them correctly in the terminal. -- Xiubo Vlad On 4/18/22 23:52, Jos Collin wrote: Do you have mounted clients? How many clients do you have? Please see: https://tracker.ceph.com/issues/55197 On 19/04/22 01:13, Vladimir Brik wrote: Does anybody know why cephfs-top may only display header lines (date, client types, metric names) but no actual data? When I run it, cephfs-top consumes quite a bit of the CPU and generates quite a bit of network traffic, but it doesn't actually display the data. I poked around in the source code and it seems like it might be curses issue, but I am not sure. Vlad ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: df shows wrong size of cephfs share when a subdirectory is mounted
On Tue, Apr 19, 2022 at 08:51:50PM +, Ryan Taylor wrote:
> Thanks for the pointers! It does look like
> https://tracker.ceph.com/issues/55090
> and I am not surprised Dan and I are hitting the same issue...
Just a wild guess (already asked this on the tracker):
Is it possible that you're using different credentials/keys so that the
credentials used for mounting the subdir are not allowed to access the
volume base directory? Would it be possible to get more details on the
two mount commands being used?
Cheers,
--
Luís
>
>
> I am using the latest available Almalinux 8, 4.18.0-348.20.1.el8_5.x86_64
>
> Installing kernel-debuginfo-common-x86_64
> I see in
> /usr/src/debug/kernel-4.18.0-348.2.1.el8_5/linux-4.18.0-348.2.1.el8_5.x86_64/fs/ceph/quota.c
> for example:
>
> static inline bool ceph_has_realms_with_quotas(struct inode *inode)
> {
> struct super_block *sb = inode->i_sb;
> struct ceph_mds_client *mdsc = ceph_sb_to_mdsc(sb);
> struct inode *root = d_inode(sb->s_root);
>
> if (atomic64_read(&mdsc->quotarealms_count) > 0)
> return true;
> /* if root is the real CephFS root, we don't have quota realms */
> if (root && ceph_ino(root) == CEPH_INO_ROOT)
> return false;
> /* otherwise, we can't know for sure */
> return true;
> }
>
> So this EL8.5 kernel already has at least some of the patches from
> https://lore.kernel.org/all/20190301175752.17808-1-lhenriq...@suse.com/T/#u
> for https://tracker.ceph.com/issues/38482
> That does not mention a specific commit, just says "Merged into 5.2-rc1."
>
> So it seems https://tracker.ceph.com/issues/55090 is either a new issue or a
> regression of the previous issue.
>
> Thanks,
> -rt
>
> Ryan Taylor
> Research Computing Specialist
> Research Computing Services, University Systems
> University of Victoria
>
>
> From: Hendrik Peyerl
> Sent: April 19, 2022 6:05 AM
> To: Ramana Venkatesh Raja
> Cc: Ryan Taylor; ceph-users@ceph.io
> Subject: Re: [ceph-users] df shows wrong size of cephfs share when a
> subdirectory is mounted
>
> Notice: This message was sent from outside the University of Victoria email
> system. Please be cautious with links and sensitive information.
>
>
> I did hit this issue aswell: https://tracker.ceph.com/issues/38482
>
> you will need a kernel >= 5.2 that can handle the quotas on subdirectories.
>
>
> > On 19. Apr 2022, at 14:47, Ramana Venkatesh Raja wrote:
> >
> > On Sat, Apr 16, 2022 at 10:15 PM Ramana Venkatesh Raja
> > wrote:
> >>
> >> On Thu, Apr 14, 2022 at 8:07 PM Ryan Taylor wrote:
> >>>
> >>> Hello,
> >>>
> >>>
> >>> I am using cephfs via Openstack Manila (Ussuri I think).
> >>>
> >>> The cephfs cluster is v14.2.22 and my client has kernel
> >>> 4.18.0-348.20.1.el8_5.x86_64
> >>>
> >>>
> >>> I have a Manila share
> >>>
> >>> /volumes/_nogroup/55e46a89-31ff-4878-9e2a-81b4226c3cb2
> >>>
> >>>
> >>> that is 5000 GB in size. When I mount it the size is reported correctly:
> >>>
> >>>
> >>> # df -h /cephfs
> >>> Filesystem
> >>> Size Used Avail Use% Mounted on
> >>> 10.30.201.3:6789,10.30.202.3:6789,10.30.203.3:6789:/volumes/_nogroup/55e46a89-31ff-4878-9e2a-81b4226c3cb2
> >>> 4.9T 278G 4.7T 6% /cephfs
> >>>
> >>>
> >>> However when I mount a subpath /test1 of my share, then both the size and
> >>> usage are showing the size of the whole cephfs filesystem rather than my
> >>> private share.
> >>>
> >>>
> >>> # df -h /cephfs
> >>> Filesystem
> >>>Size Used Avail Use% Mounted on
> >>> 10.30.201.3:6789,10.30.202.3:6789,10.30.203.3:6789:/volumes/_nogroup/55e46a89-31ff-4878-9e2a-81b4226c3cb2/test1
> >>> 4.0P 277T 3.7P 7% /cephfs
> >>>
> >>
> >> What are the capabilities of the ceph client user ID that you used to
> >> mount "/volumes/_nogroup/55e46a89-31ff-4878-9e2a-81b4226c3cb2/test1" ?
> >> Maybe you're hitting this limitation in
> >> https://docs.ceph.com/en/latest/cephfs/quota/#limitations ,
> >> "Quotas must be configured carefully when used with path-based mount
> >> restrictions. The client needs to have access to the directory inode
> >> on which quotas are configured in order to enforce them. If the client
> >> has restricted access to a specific path (e.g., /home/user) based on
> >> the MDS capability, and a quota is configured on an ancestor directory
> >> they do not have access to (e.g., /home), the client will not enforce
> >> it. When using path-based access restrictions be sure to configure the
> >> quota on the directory the client is restricted too (e.g., /home/user)
> >> or something nested beneath it. "
> >>
> >
> > Hi Ryan,
> >
> > I think you maybe actually hitting this
> > https://tracker.ceph.com/issues/55090 . Are you facing this issue with
[ceph-users] Monitor doesn't start anymore...
Hi Ceph users,
after a long time without any major incident (which is great for such a
complex piece of software!), I've finally encountered a problem with our
Ceph installation. All of a sudden the monitor service on one of the
nodes doesn't start anymore. It crashes immediately when I try to start
the service. See below for the journalctl log. I found some
recommendations on the web to check whether the keys/config are all
identical but it seems they are and I actually didn't change anything.
It just stopped working and now it won't run anymore. What can I do to
further diagnose this issue?
Thank you / Best regards
Ranjan
Apr 12 14:58:49 stage2 ceph-mon[614074]: 2022-04-12T14:58:49.652+0200
7fa0c67fc640 -1 mon.stage2@-1(???) e3 handle_auth_bad_method hmm, they
didn't like 2 result (13) Permission denied
Apr 12 14:58:49 stage2 ceph-mon[614074]: 2022-04-12T14:58:49.856+0200
7fa0c67fc640 -1 mon.stage2@1(probing) e3 handle_auth_bad_method hmm,
they didn't like 2 result (13) Permission denied
Apr 12 14:58:49 stage2 ceph-mon[614074]: [259B blob data]
Apr 12 14:58:49 stage2 ceph-mon[614074]: PutCF( prefix = mon_sync key =
'latest_monmap' value size = 454)
Apr 12 14:58:49 stage2 ceph-mon[614074]: PutCF( prefix = mon_sync key =
'in_sync' value size = 8)
Apr 12 14:58:49 stage2 ceph-mon[614074]: PutCF( prefix = mon_sync key =
'last_committed_floor' value size = 8)
Apr 12 14:58:49 stage2 ceph-mon[614074]: ./src/mon/MonitorDBStore.h: In
function 'int
MonitorDBStore::apply_transaction(MonitorDBStore::TransactionRef)'
thread 7fa0a1ffb640 time 2022-04-12T>
Apr 12 14:58:49 stage2 ceph-mon[614074]: ./src/mon/MonitorDBStore.h:
351: ceph_abort_msg("failed to write to db")
Apr 12 14:58:49 stage2 ceph-mon[614074]: ceph version 16.2.7
(dd0603118f56ab514f133c8d2e3adfc983942503) pacific (stable)
Apr 12 14:58:49 stage2 ceph-mon[614074]: 1: (ceph::__ceph_abort(char
const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd7)>
Apr 12 14:58:49 stage2 ceph-mon[614074]: 2:
(MonitorDBStore::apply_transaction(std::shared_ptr)+0x890)
[0x5636462d9cc0]
Apr 12 14:58:49 stage2 ceph-mon[614074]: 3:
(Monitor::sync_start(entity_addrvec_t&, bool)+0x370) [0x563646294020]
Apr 12 14:58:49 stage2 ceph-mon[614074]: 4:
(Monitor::handle_probe_reply(boost::intrusive_ptr)+0xa2a)
[0x56364629e2ea]
Apr 12 14:58:49 stage2 ceph-mon[614074]: 5:
(Monitor::handle_probe(boost::intrusive_ptr)+0x2ff)
[0x56364629f8cf]
Apr 12 14:58:49 stage2 ceph-mon[614074]: 6:
(Monitor::dispatch_op(boost::intrusive_ptr)+0xe9f)
[0x5636462ace9f]
Apr 12 14:58:49 stage2 ceph-mon[614074]: 7:
(Monitor::_ms_dispatch(Message*)+0x411) [0x5636462ad651]
Apr 12 14:58:49 stage2 ceph-mon[614074]: 8:
(Dispatcher::ms_dispatch2(boost::intrusive_ptr const&)+0x5d)
[0x5636462db5ad]
Apr 12 14:58:49 stage2 ceph-mon[614074]: 9:
(Messenger::ms_deliver_dispatch(boost::intrusive_ptr
const&)+0x450) [0x7fa0d0b1d3a0]
Apr 12 14:58:49 stage2 ceph-mon[614074]: 10:
(DispatchQueue::entry()+0x647) [0x7fa0d0b1a757]
Apr 12 14:58:49 stage2 ceph-mon[614074]: 11:
(DispatchQueue::DispatchThread::entry()+0x11) [0x7fa0d0bdb451]
Apr 12 14:58:49 stage2 ceph-mon[614074]: 12:
/lib/x86_64-linux-gnu/libc.so.6(+0x94947) [0x7fa0d0147947]
Apr 12 14:58:49 stage2 ceph-mon[614074]: 13: clone()
Apr 12 14:58:49 stage2 ceph-mon[614074]: 2022-04-12T14:58:49.864+0200
7fa0a1ffb640 -1 ./src/mon/MonitorDBStore.h: In function 'int
MonitorDBStore::apply_transaction(MonitorDBStore::Transact>
Apr 12 14:58:49 stage2 ceph-mon[614074]: ./src/mon/MonitorDBStore.h:
351: ceph_abort_msg("failed to write to db")
Apr 12 14:58:49 stage2 ceph-mon[614074]: ceph version 16.2.7
(dd0603118f56ab514f133c8d2e3adfc983942503) pacific (stable)
Apr 12 14:58:49 stage2 ceph-mon[614074]: 1: (ceph::__ceph_abort(char
const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd7)>
Apr 12 14:58:49 stage2 ceph-mon[614074]: 2:
(MonitorDBStore::apply_transaction(std::shared_ptr)+0x890)
[0x5636462d9cc0]
Apr 12 14:58:49 stage2 ceph-mon[614074]: 3:
(Monitor::sync_start(entity_addrvec_t&, bool)+0x370) [0x563646294020]
Apr 12 14:58:49 stage2 ceph-mon[614074]: 4:
(Monitor::handle_probe_reply(boost::intrusive_ptr)+0xa2a)
[0x56364629e2ea]
Apr 12 14:58:49 stage2 ceph-mon[614074]: 5:
(Monitor::handle_probe(boost::intrusive_ptr)+0x2ff)
[0x56364629f8cf]
===
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Slow read write operation in ssd disk pool
Hi, I will check and confirm the lebel. In the mean time can you guys help me to find out the root cause of this issue, and how I can resolve this? Is there any ceph configuration issue or anything we should check Please advise. Regards, Munna On Wed, Apr 20, 2022 at 7:29 PM Marc wrote: > > > > Those are not enterprise SSD, Samsung labeled and all are 2TB in size. we > > have three node, 12 disk on each node > > > > I think you should use the drives labeled with the lower case 's' of > samsung. > > > > > ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Monitor doesn't start anymore...
Hi Stefan, thanks for the hint. Yes, the mon store is owned by user ceph. It's a normal folder under /var/lib/ceph/mon/... and there are various files in there - all with owner ceph:ceph. Looks normal to me (like on the other nodes). I've read the docs you provided and I wonder which of those steps are actually necessary because I obviously have many of those things already in place. I guess the most important part is step 15, right? ceph-mon --mkfs -i stage2 --monmap /tmp/monmap --keyring /tmp/ceph.mon.keyring So I would completely delete the /var/lib/ceph/mon/ceph-stage2 folder and recreate it as empty, because the files in there have probably become corrupted, correct? Then I would run the above command and it would initialize a new monitor automatically in that folder by default? I don't have to specify the exact path on the command-line - it is automatically determined by the -i argument, right? So, I only need the keyring and the monmap. How do I get the monmap? I tried to run: ceph-mon -i stage2 --extract-monmap /tmp/monmap And I get some binary stuff written to /tmp/monmap. Is that a decent monmap? Is there also a way to extract the keyring? There's a keyring file in /var/lib/ceph/mon/ceph-stage2. Can I just use that one? Thank you / BR Ranjan Stefan Kooman schrieb am 20.04.2022 um 17:01: > On 4/20/22 16:52, Ranjan Ghosh wrote: >> Hi Ceph users, >> >> after a long time without any major incident (which is great for such a >> complex piece of software!), I've finally encountered a problem with our >> Ceph installation. All of a sudden the monitor service on one of the >> nodes doesn't start anymore. It crashes immediately when I try to start >> the service. See below for the journalctl log. I found some >> recommendations on the web to check whether the keys/config are all >> identical but it seems they are and I actually didn't change anything. >> It just stopped working and now it won't run anymore. What can I do to >> further diagnose this issue? > > > Is the mon store stored on a separate disk? Is the mon store owned by > user ceph? Any possible filesystem / hardware issues on the system? > > If not, you might want to replace the mon store with a new one with a > new one [1]. > > Gr. Stefan > > [1]: https://docs.ceph.com/en/latest/install/manual-deployment/ ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Monitor doesn't start anymore...
Hi Stefan, amazing. You're a genius. It worked. Everything back in order. Wonderful. These things are perhaps not so exciting for all of you Ceph experts but it's certainly *very* exciting for me to do stuff like this on a live system. *Phew, sweat*. So happy this is finally resolved and nobody noticed there was even a problem ;-) 1000 thanks!!! Ranjan Stefan Kooman schrieb am 20.04.2022 um 17:50: > On 4/20/22 17:36, Ranjan Ghosh wrote: >> Hi Stefan, >> >> thanks for the hint. Yes, the mon store is owned by user ceph. It's a >> normal folder under /var/lib/ceph/mon/... and there are various files >> in there - all with owner ceph:ceph. Looks normal to me (like on the >> other nodes). >> I've read the docs you provided and I wonder which of those steps are >> actually necessary because I obviously have many of those things >> already in place. >> I guess the most important part is step 15, right? >> >> ceph-mon --mkfs -i stage2 --monmap /tmp/monmap --keyring >> /tmp/ceph.mon.keyring >> >> So I would completely delete the /var/lib/ceph/mon/ceph-stage2 folder >> and recreate it as empty, because the files in there have probably >> become corrupted, correct? > > Yeah, that's what I suspect. > >> Then I would run the above command and it would initialize a new >> monitor automatically in that folder by default? I don't have to >> specify the exact path on the command-line - it is automatically >> determined by the -i argument, right? So, I only need the keyring >> and the monmap. How do I get the monmap? I tried to run: >> >> ceph-mon -i stage2 --extract-monmap /tmp/monmap >> >> And I get some binary stuff written to /tmp/monmap. Is that a decent >> monmap? >> Is there also a way to extract the keyring? There's a keyring file in >> /var/lib/ceph/mon/ceph-stage2. Can I just use that one? > > Here my notes from a re-install mon test I did. I susbstituted the > name with "stage2": > > chown -R ceph.ceph /var/lib/ceph/mon > sudo -u ceph ceph auth get mon. -o /tmp/keyring > exported keyring for mon. > sudo -u ceph ceph mon getmap -o /tmp/monmap.map > got monmap epoch 8 > chown -R ceph:ceph /var/lib/ceph/mon/ceph-stage2 > sudo -u ceph ceph-mon -i stage2 --mkfs --monmap /tmp/monmap.map > --keyring /tmp/keyring --setuser ceph --setgroup ceph > > systemctl enable ceph-mon@stage2.service > > If you have the mgr co-located on the monitor node, you need to make > sure you do below for the manager: > > mkdir /var/lib/ceph/mgr/ceph-stage2 > ceph auth get mgr.stage2 >> /var/lib/ceph/mgr/ceph-stage2/keyring > chown -R ceph.ceph /var/lib/ceph/mgr/ > > systemctl enable ceph-mgr@stage2.service > > After a reboot the mon and the mgr should come online. > > Gr. Stefan ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: v17.2.0 Quincy released
On Wed, Apr 20, 2022 at 7:22 AM Stefan Kooman wrote: > > On 4/20/22 03:36, David Galloway wrote: > > We're very happy to announce the first stable release of the Quincy series. > > > > We encourage you to read the full release notes at > > https://ceph.io/en/news/blog/2022/v17-2-0-quincy-released/ > > When upgrading a MDS to 16.2.7 in a non cephadm environment you should > set "ceph config set mon mon_mds_skip_sanity 1". And after the upgrade > remove it. Yes! > I'm wondering if the same step is needed when upgrading from Octopus > (and or pacific < 16.2.7) to Quincy? It's not in the release notes, but > just double checking here [1]. Yes it is necessary. -- Patrick Donnelly, Ph.D. He / Him / His Principal Software Engineer Red Hat, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] replaced osd's get systemd errors
I added some osd's which are up and running with: ceph-volume lvm create --data /dev/sdX --dmcrypt But I am still getting such messages of the newly created osd's systemd: Job dev-disk-by\x2duuid-7a8df80d\x2d4a7a\x2d469f\x2d868f\x2d8fd9b7b0f09d.device/start timed out. systemd: Timed out waiting for device dev-disk-by\x2duuid-7a8df80d\x2d4a7a\x2d469f\x2d868f\x2d8fd9b7b0f09d.device. systemd: Dependency failed for /var/lib/ceph/osd/ceph-11. systemd: Job dev-disk-by\x2duuid-864b01aa\x2d1abf\x2d4dc0\x2da532\x2dced7cb321f4a.device/start timed out. systemd: Timed out waiting for device dev-disk-by\x2duuid-864b01aa\x2d1abf\x2d4dc0\x2da532\x2dced7cb321f4a.device. systemd: Dependency failed for /var/lib/ceph/osd/ceph-1. ceph 14.2.22 ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Ceph Multisite Cloud Sync Module
My apologies if my previous message seemed negative to your docs in any way. They are very good and work as documented. The area that tripped me up originally and caused confusion on my part is the Ceph docs that do not state clearly that one needs a separate radosgw daemon for any of the Sync Modules. The additional ragdosgw process is used to host the zone that specifies the sync tier and can be used in sync policies. This of course means you need to run multiple radosgw process on your host set. I am somewhat new to multisite and my initial setup was associating one zone to one cluster. My mind was stuck in this paradigm. Once I got unstuck from this way of thinking I re read the post and it does indeed make sense and works as intended. Thanks for taking the time to write your post. Again apologies -- the confusion was my end. -- Mark Selby Sr Linux Administrator, The Voleon Group mse...@voleon.com This email is subject to important conditions and disclosures that are listed on this web page: https://voleon.com/disclaimer/. On 4/19/22, 1:40 AM, "Etienne Menguy" wrote: Hey, I wrote this blog post, which part is unclear? It should 'just work'. Étienne -Original Message- From: Mark Selby Sent: mardi 19 avril 2022 06:17 To: ceph-users@ceph.io Subject: [ceph-users] Ceph Multisite Cloud Sync Module I am trying to get the Ceph Multisite Clous Sync module working with Amazon S3. The docs are not clear on how the sync module is actually configured. I just want a POC of the most simple config. Can anyone share the config and radosgw-admin commands that were invoked to create a simple sync setup. The closest docs that I have seen are https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcroit.io%2Fblog%2Fsetting-up-ceph-cloud-sync-module&data=04%7C01%7Cmselby%40voleon.com%7C30de90f1703145ce01f708da21e047ad%7C45212fd85f544a19a6ba493ff6e072b1%7C0%7C0%7C637859544416809376%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=hcnF6NQxz5Mtd0T0c%2FuJz%2Ff2wkv%2BCi%2F0oIpPtAkq1EM%3D&reserved=0 and to be honest they do not make a lot of sense. Thanks! -- Mark Selby Sr Linux Administrator, The Voleon Group mse...@voleon.com ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] RGW limiting requests/sec
Hello. I have a Ceph cluster (using Nautilus) in a lab environment on a smaller scale than the production environment. We had some problems with timeouts in production, so I started doing some benchmarking tests in this lab environment. The problem is that the performance of RGW (with beast) is very low, I'm only getting around 600 requests/s using "wrk" making HEAD requests. The size of the RGWs VMs are the same in both lab and production. Do you have any idea what could be causing this limit? I tried to increase the rgw thread pool size but all it did was decrease the number of requests/sec. My rgw client configuration: [client.rgw.ceph-rgw-1.rgw0] host = ceph-rgw-1 keyring = /var/lib/ceph/radosgw/ceph-rgw.ceph-rgw-1.rgw0/keyring log_to_file = true log file = /var/log/ceph/ceph-rgw-ceph-rgw-1.rgw0.log rgw frontends = beast endpoint=10.79.35.245:8080 Another test I did was to switch to civitweb, and the number of requests/sec increased to 1800, which I found strange because I thought beast would be more efficient. To discard network problem, I started an nginx, and in that I managed to reach 16 requests/sec. Am I missing something here? Thank you very much, Marcelo. "Essa mensagem do SERVIÇO FEDERAL DE PROCESSAMENTO DE DADOS (SERPRO), empresa pública federal regida pelo disposto na Lei Federal nº 5.615, é enviada exclusivamente ao destinatário informado e pode conter dados pessoais, protegidos pela Lei Geral de Proteção de Dados (Lei 13.709/2018), assim como informações confidenciais, protegidas por sigilo profissional. O SERPRO ressalta seu comprometimento em assegurar a segurança e a proteção das informações contidas neste e-mail e informa que a sua utilização desautorizada é ilegal e sujeita o infrator às penas da lei. Se você o recebeu indevidamente, queira, por gentileza, reenviá-lo ao emitente, esclarecendo o equívoco." "This message from SERVIÇO FEDERAL DE PROCESSAMENTO DE DADOS (SERPRO) - a government company established under Brazilian law (5.615/70) - is directed exclusively to its addressee and may contain personal data protected by the General Data Protection Law (13.709/2018) as well as confidencial data, protected under professional secrecy rules. SERPRO highlights its commitment to ensuring the security and protection of the information contained in this email and its unauthorized use is illegal and may subject the transgressor to the law´s penalties. If you´re not the addressee, please send it back, elucidating the failure." ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Ceph Multisite Cloud Sync Module
Thanks very much for responding - my problem was not realizing that you need a
separate ragdosgw daemon for the zone that hosts the cloud tier. Once I got
past this concept your example worked great for a simple setup.
--
Mark Selby
Sr Linux Administrator, The Voleon Group
mse...@voleon.com
This email is subject to important conditions and disclosures that are listed
on this web page: https://voleon.com/disclaimer/.
On 4/19/22, 1:26 AM, "Soumya Koduri" wrote:
Hi,
On 4/19/22 09:47, Mark Selby wrote:
> I am trying to get the Ceph Multisite Clous Sync module working with
Amazon S3. The docs are not clear on how the sync module is actually
configured. I just want a POC of the most simple config. Can anyone share the
config and radosgw-admin commands that were invoked to create a simple sync
setup. The closest docs that I have seen are
https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcroit.io%2Fblog%2Fsetting-up-ceph-cloud-sync-module&data=04%7C01%7Cmselby%40voleon.com%7C7eeea0b9c0564670c7b008da21de423e%7C45212fd85f544a19a6ba493ff6e072b1%7C0%7C0%7C637859535743326124%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=ne0jVeBwpij1xC1LjM9JsPpX2ASXvoJPU7GmS%2BNraUM%3D&reserved=0
and to be honest they do not make a lot of sense.
>
>
I presume (from your last email) that you are familiar with setting up
multisite between zones in a single zonegroup [1] . Cloud sync zone is
similar to creating any other zone [2] but with additional option
'--tier-type=cloud' specified. Below are the minimal steps you may need
to get started with it -
1) To create cloud sync zone-
# radosgw-admin zone create --tier-type=cloud --rgw-zone=
--rgw-zonegroup= --rgw-realm=
--access-key= --secret=
--endpoints=http://{fqdn}:80
eg., radosgw-admin zone create --rgw-zone=cloud-zone
--rgw-zonegroup=default --rgw-realm=realm_default --tier-type=cloud
--access-key= --secret=
--endpoints=http://localhost:8002
2) Update cloud zone with the remote endpoint connection details -
# radosgw-admin zone modify --rgw-zone=
--rgw-zonegroup= --rgw-realm= \
--tier-config=connection.access_key=,connection.secret=,connection.endpoint=,connection.host_style=path,acls.source_id=,acls.dest_id=,target-path=
note: host_style, acls and target-path are optional. If not specified,
the zone will create a default bucket on the cloud endpoint to sync
entries to and maps the same rgw user to remote endpoint.
eg., radosgw-admin zone modify --rgw-zone=cloud-zone
--rgw-zonegroup=default --rgw-realm=realm_default
--tier-config=connection.access_key=L#R,connection.secret=Y##t,connection.endpoint=https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2F10.xx.xx.xx%3A80%2F&data=04%7C01%7Cmselby%40voleon.com%7C7eeea0b9c0564670c7b008da21de423e%7C45212fd85f544a19a6ba493ff6e072b1%7C0%7C0%7C637859535743326124%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=Zz46bUMcoOP8%2F%2F1Gt%2BlaFOAj39l8TCrYDv7XNXqRs3I%3D&reserved=0,acls.source_id=testid,acls.dest_id=aws-user,target_path=cloud-sync
3) radosgw-admin period update --commit
4) radosgw-admin sync status // to fetch sync status across zones
including cloud sync zone.
Hope this helps!
-Soumya
[1]
https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2Fradosgw%2Fmultisite%2F%23multisite&data=04%7C01%7Cmselby%40voleon.com%7C7eeea0b9c0564670c7b008da21de423e%7C45212fd85f544a19a6ba493ff6e072b1%7C0%7C0%7C637859535743326124%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=biHFzyoJ8y2YnODV9c9ZBKBbUqTBkC%2FO2O3kOAOjtMk%3D&reserved=0
[2]
https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2Fradosgw%2Fcloud-sync-module%2F&data=04%7C01%7Cmselby%40voleon.com%7C7eeea0b9c0564670c7b008da21de423e%7C45212fd85f544a19a6ba493ff6e072b1%7C0%7C0%7C637859535743326124%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=gURadkbDu%2BRhxNJSEEl6Tdu88J6xgpUMvubp3unR%2Bl0%3D&reserved=0
>
>
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Ceph RGW Multisite Multi Zonegroup Build Problems
I rebuilt the setup from scratch and left off --master from the zones in the other zonegroups and it had no effect on the outcome. ulrich.kl...@ulrichklein.net has tried this as well and sees the same failure. I hope there is someone who can see what we are doing wrong. This topology is shown here https://events.static.linuxfound.org/sites/events/files/slides/Linuxcon_multisitev2.pdf which is an official redhat doc. -- Mark Selby Sr Linux Administrator, The Voleon Group mse...@voleon.com This email is subject to important conditions and disclosures that are listed on this web page: https://voleon.com/disclaimer/. On 4/19/22, 1:59 AM, "Eugen Block" wrote: Hi, unless there are copy/paste mistakes involved I believe you shouldn't specify '--master' for the secondary zone because you did that already for the first zone which is supposed to be the master zone. You specified '--rgw-zone=us-west-1' as the master zone within your realm, but then you run this command on the second cluster: radosgw-admin zone create --rgw-zone=eu-west-1 \ --rgw-zonegroup=eu \ --default \ --master \ --endpoints=https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fceph2dev01.acme.com%2F&data=04%7C01%7Cmselby%40voleon.com%7C5a448eef012a44477c0908da21e28ac3%7C45212fd85f544a19a6ba493ff6e072b1%7C0%7C0%7C637859555823860934%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=AOAAqwVsdgCZZT3uRiLcIgpR5swszqo3JrCbT4dAowA%3D&reserved=0 That's the reason for this error when trying to commit on the second cluster: 2022-04-16T09:16:20.345-0700 7faf98ab6380 1 Cannot find zone id=9f8a06eb-5a1c-4052-b04d-359f21c95371 (name=eu-west-1), switching to local zonegroup configuration Because your master zone is this one: "master_zone": "d7ceaa4f-06c0-4c21-bcec-efe90f55ecfd" I'd recommend to purge the secondary zone and start over. If this is not the root cause, please update your post. Regards, Eugen Zitat von Mark Selby : > I have been trying to build a multisite ceph rgw installation with a > single realm, multiple zonegroups, and a single zone per zonegroup. This > is a model around 3 locations spread over long distances. I have > sucessfully create an installation with a single realm, a single > zonegroup and multiple zones on that one zonegroup. > > I have had no luck getting my multiple zonegroup installation even > setup. I have read the docs over and over but I still think that I am > doing something incorrect (or possibly a bug?) > > I am running Pacific 16.2.7 in a containerized environment > > I have created a github gist of all of the commands and output show > below as that may be easier to read for some. > > https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgist.github.com%2Ftokenrain%2F4edf85b0060ce5004f2003aa8a66e67d&data=04%7C01%7Cmselby%40voleon.com%7C5a448eef012a44477c0908da21e28ac3%7C45212fd85f544a19a6ba493ff6e072b1%7C0%7C0%7C637859555823860934%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=wt%2BJsyiA0GjlwHgmqHaM7eV1Y4D9Y6Ci8uJm%2BxBb6nI%3D&reserved=0 > > Cluster 1 and Cluster 2 are separate ceph clusters. Cluster 1 commands > were run on a node in cluster1 and Cluster 2 commands were run on a node > in cluster2 > > All and any help is greatly appreciated. > > > # TOPOLOGY # > > > realm = acme.com > zonegroup = us > zone = us-west-1 > zonegroup = eu > zone = eu-west-1 > zonegroup = as > zone = as-west-1 > > ## > # CLUSTER 1 COMMANDS # > ## > > radosgw-admin realm create --rgw-realm=acme --default > > radosgw-admin zonegroup create --rgw-zonegroup=us --rgw-realm=acme > --master --default --endpoints=https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fceph1dev01.acme.com%2F&data=04%7C01%7Cmselby%40voleon.com%7C5a448eef012a44477c0908da21e28ac3%7C45212fd85f544a19a6ba493ff6e072b1%7C0%7C0%7C637859555823860934%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=ojLgv0XS7mQ49UVuZtYZ8SZMSBjbi75xCkeR1i3m3EE%3D&reserved=0 > > radosgw-admin zone create --rgw-zonegroup=us \ > --rgw-zone=us-west-1 \ > --master \ > --default \ > --endpoints=https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fceph1dev01.acme.com%2F&data=04%7C01%7Cmselby%40voleon.com%7C5a448eef012a44477c0908da21e28ac3%7C45212fd85f544a19a6ba493ff6e07
[ceph-users] Re: df shows wrong size of cephfs share when a subdirectory is mounted
Hi Luís,
The same cephx key is used for both mounts. It is a regular rw key which does
not have permission to set any ceph xattrs (that was done separately with a
different key).
But it can read ceph xattrs and set user xattrs.
I just did a test using the latest Fedora 35 kernel and reproduced the problem:
[fedora@cephtest ~]$ sudo mkdir /mnt/ceph1
[fedora@cephtest ~]$ sudo mkdir /mnt/ceph2
[fedora@cephtest ~]$ sudo mount -t ceph
10.30.201.3:6789,10.30.202.3:6789,10.30.203.3:6789:/volumes/_nogroup/55e46a89-31ff-4878-9e2a-81b4226c3cb2
/mnt/ceph1 -o name=rwkey,secret=...
[fedora@cephtest ~]$ sudo mkdir /mnt/ceph1/testsubdir
[fedora@cephtest ~]$ sudo mount -t ceph
10.30.201.3:6789,10.30.202.3:6789,10.30.203.3:6789:/volumes/_nogroup/55e46a89-31ff-4878-9e2a-81b4226c3cb2/testsubdir
/mnt/ceph2 -o name=rwkey,secret=...
[fedora@cephtest ~]$ df | grep ceph
10.30.201.3:6789,10.30.202.3:6789,10.30.203.3:6789:/volumes/_nogroup/55e46a89-31ff-4878-9e2a-81b4226c3cb2
524288291385344 4951494656 6%
/mnt/ceph1
10.30.201.3:6789,10.30.202.3:6789,10.30.203.3:6789:/volumes/_nogroup/55e46a89-31ff-4878-9e2a-81b4226c3cb2/testsubdir
4287562399744 295238516736 3992323883008 7% /mnt/ceph2
[fedora@cephtest ~]$ uname -r
5.16.20-200.fc35.x86_64
Furthermore I then repeated my earlier test regarding ceph.quota.max_bytes.
The volume root already has the right quota based on the size of my Manila
share in Openstack, and it matches the size reported by df (5000 GiB)
[fedora@cephtest ~]$ getfattr -n ceph.quota.max_bytes /mnt/ceph1/
getfattr: Removing leading '/' from absolute path names
# file: mnt/ceph1/
ceph.quota.max_bytes="536870912"
And on a separate system with admin credentials I applied a max_bytes quota to
the testsubdir:
sudo setfattr -n ceph.quota.max_bytes -v 121212
/mnt/cephfs/volumes/_nogroup/55e46a89-31ff-4878-9e2a-81b4226c3cb2/testsubdir/
I unmounted and remounted testsubdir exactly as before, but even with
ceph.quota.max_bytes applied on the subdir it still shows the wrong size:
[fedora@cephtest ~]$ df | grep ceph
10.30.201.3:6789,10.30.202.3:6789,10.30.203.3:6789:/volumes/_nogroup/55e46a89-31ff-4878-9e2a-81b4226c3cb2
524288291385344 4951494656 6%
/mnt/ceph1
10.30.201.3:6789,10.30.202.3:6789,10.30.203.3:6789:/volumes/_nogroup/55e46a89-31ff-4878-9e2a-81b4226c3cb2/testsubdir
4287544954880 295264587776 3992280367104 7% /mnt/ceph2
[fedora@cephtest ~]$ getfattr -n ceph.quota.max_bytes /mnt/ceph1/testsubdir/
getfattr: Removing leading '/' from absolute path names
# file: mnt/ceph1/testsubdir/
ceph.quota.max_bytes="121212"
[fedora@cephtest ~]$ getfattr -n ceph.quota.max_bytes /mnt/ceph2
getfattr: Removing leading '/' from absolute path names
# file: mnt/ceph2
ceph.quota.max_bytes="121212"
Thanks,
-rt
From: Luís Henriques
Sent: April 20, 2022 7:16 AM
To: Ryan Taylor
Cc: Hendrik Peyerl; Ramana Venkatesh Raja; ceph-users@ceph.io
Subject: Re: [ceph-users] Re: df shows wrong size of cephfs share when a
subdirectory is mounted
Notice: This message was sent from outside the University of Victoria email
system. Please be cautious with links and sensitive information.
On Tue, Apr 19, 2022 at 08:51:50PM +, Ryan Taylor wrote:
> Thanks for the pointers! It does look like
> https://tracker.ceph.com/issues/55090
> and I am not surprised Dan and I are hitting the same issue...
Just a wild guess (already asked this on the tracker):
Is it possible that you're using different credentials/keys so that the
credentials used for mounting the subdir are not allowed to access the
volume base directory? Would it be possible to get more details on the
two mount commands being used?
Cheers,
--
Luís
>
>
> I am using the latest available Almalinux 8, 4.18.0-348.20.1.el8_5.x86_64
>
> Installing kernel-debuginfo-common-x86_64
> I see in
> /usr/src/debug/kernel-4.18.0-348.2.1.el8_5/linux-4.18.0-348.2.1.el8_5.x86_64/fs/ceph/quota.c
> for example:
>
> static inline bool ceph_has_realms_with_quotas(struct inode *inode)
> {
> struct super_block *sb = inode->i_sb;
> struct ceph_mds_client *mdsc = ceph_sb_to_mdsc(sb);
> struct inode *root = d_inode(sb->s_root);
>
> if (atomic64_read(&mdsc->quotarealms_count) > 0)
> return true;
> /* if root is the real CephFS root, we don't have quota realms */
> if (root && ceph_ino(root) == CEPH_INO_ROOT)
> return false;
> /* otherwise, we can't know for sure */
> return true;
> }
>
> So this EL8.5 kernel already has at least some of the patches from
> https://lore.kernel.org/all/20190301175752.17808-1-lhenriq...@suse.com/T/#u
> for https://tracker.ceph.com/issues/38482
> That does not mention a specific commit, just says "Merged into 5.2-rc1."
>
> So it seems https://tracker.ceph.com/issues/55090 is either a new issue or a
> regressio
[ceph-users] No Ceph User + Dev Monthly Meetup this month
Hi everyone, This month's Ceph User + Dev Monthly Meetup has been canceled due to the ongoing Ceph Developer Summit. However, we'd like to know if there is any interest in an APAC friendly meeting. If so, we could alternate between APAC and EMEA friendly meetings, like we do for CDM. Thanks, Neha ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] How to build custom binary?
Hi everyone, I’m trying to build my own Ceph .deb packages to solve the issue with this bug https://tracker.ceph.com/issues/51327 in the Pacific release but, after building all the .deb packages, I’m not able to install them on my rgw.. Thank you, Fabio Pasetti ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Ceph octopus v15.2.15-20220216 status
Does the v15.2.15-20220216 container include backports published since the release of v15.2.15-20211027 ? I'm interested in BACKPORT #53392 https://tracker.ceph.com/issues/53392, which was merged into the ceph:octopus branch on February 10th. ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: RGW limiting requests/sec
Hi, RGW rate-limits available starting from Quincy release k Sent from my iPhone > On 20 Apr 2022, at 20:58, Marcelo Mariano Miziara > wrote: > > Hello. I have a Ceph cluster (using Nautilus) in a lab environment on a > smaller scale than the production environment. We had some problems with > timeouts in production, so I started doing some benchmarking tests in this > lab environment. The problem is that the performance of RGW (with beast) is > very low, I'm only getting around 600 requests/s using "wrk" making HEAD > requests. > > The size of the RGWs VMs are the same in both lab and production. > > Do you have any idea what could be causing this limit? I tried to increase > the rgw thread pool size but all it did was decrease the number of > requests/sec. > > My rgw client configuration: > [client.rgw.ceph-rgw-1.rgw0] > host = ceph-rgw-1 > keyring = /var/lib/ceph/radosgw/ceph-rgw.ceph-rgw-1.rgw0/keyring > log_to_file = true > log file = /var/log/ceph/ceph-rgw-ceph-rgw-1.rgw0.log > rgw frontends = beast endpoint=10.79.35.245:8080 > > Another test I did was to switch to civitweb, and the number of requests/sec > increased to 1800, which I found strange because I thought beast would be > more efficient. > > To discard network problem, I started an nginx, and in that I managed to > reach 16 requests/sec. > > Am I missing something here? > > Thank you very much, > Marcelo. > > > "Essa mensagem do SERVIÇO FEDERAL DE PROCESSAMENTO DE DADOS (SERPRO), empresa > pública federal regida pelo disposto na Lei Federal nº 5.615, é enviada > exclusivamente ao destinatário informado e pode conter dados pessoais, > protegidos pela Lei Geral de Proteção de Dados (Lei 13.709/2018), assim como > informações confidenciais, protegidas por sigilo profissional. O SERPRO > ressalta seu comprometimento em assegurar a segurança e a proteção das > informações contidas neste e-mail e informa que a sua utilização > desautorizada é ilegal e sujeita o infrator às penas da lei. Se você o > recebeu indevidamente, queira, por gentileza, reenviá-lo ao emitente, > esclarecendo o equívoco." > "This message from SERVIÇO FEDERAL DE PROCESSAMENTO DE DADOS (SERPRO) - a > government company established under Brazilian law (5.615/70) - is directed > exclusively to its addressee and may contain personal data protected by the > General Data Protection Law (13.709/2018) as well as confidencial data, > protected under professional secrecy rules. SERPRO highlights its commitment > to ensuring the security and protection of the information contained in this > email and its unauthorized use is illegal and may subject the transgressor to > the law´s penalties. If you´re not the addressee, please send it back, > elucidating the failure." > ___ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
