> On Jan 10, 2025, at 7:46 AM, Olaf Seibert <o.seib...@syseleven.de> wrote: > > Hi! I am trying to find if Ceph has any QoS settings that apply per-client. I > would like to be able, say, to have different QoS settings for RBD clients > named "nova" and "cinder" and different again from an RGW client named > "enduser". > > While researching this, I found amongst other things, the following > references, which do not appear to be what I want: > > - https://docs.ceph.com/en/reef/dev/osd_internals/mclock_wpq_cmp_study/ and > https://docs.ceph.com/en/reef/rados/configuration/osd-config-ref/#dmclock-qos > . This appears to be instead about the balancing of user-iops against > ceph-internal iops, not balancing clients against each other.
Yes, mclock is within the cluster. > (aside: somebody should rewrite this paragraph, I find it totally > incomprehensible: > https://docs.ceph.com/en/reef/rados/configuration/osd-config-ref/#subtleties-of-mclock > ) If I could understand mblock I’d rewrite it myself :-/. Please do enter a tracker ticket. > - https://docs.ceph.com/en/latest/rbd/rbd-config-ref/#qos-settings and > https://tracker.ceph.com/issues/36191 This seems to be about setting limits > on the RBD pools as a whole, or on separate images inside them. However > separate clients each still get to use up to the limits. That link https://docs.ceph.com/en/reef/rbd/rbd-config-ref/#qos-settings does have a section that describes per-image (volume) settings, which you should be able to enforce on the OpenStack side. OpenStack / libvirt do have their own IOPS and throughput throttles you can find in their docs. This is important to do so that instances don’t DoS each other or the entire Nova node. Be sure to increase the system-wide file limit on those nodes, to like 4 million. Each RBD attachment needs two sockets to *each* OSD node for each attachment. For RGW you can do rate limiting: https://docs.ceph.com/en/latest/radosgw/adminops/#rate-limit Which I *think* is per-RGW granularity. > Is there anything else that I have not found so far but which is about > balancing individual clients across all services? Remember that a CephFS mount, an RBD attachment, and an RGW session are three different clients from Ceph’s perspective. If you actually mean rate limiting > > > -- > Olaf Seibert > Site Reliability Engineer > > SysEleven GmbH > Boxhagener Straße 80 > 10245 Berlin > > T +49 30 233 2012 0 > F +49 30 616 7555 0 > > https://www.syseleven.de > https://www.linkedin.com/company/syseleven-gmbh/ > > Current system status always at: > https://www.syseleven-status.net/ > > Company headquarters: Berlin > Registered court: AG Berlin Charlottenburg, HRB 108571 Berlin > Managing directors: Andreas Hermann, Jens Ihlenfeld, Norbert Müller, Jens > Plogsties > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
