HI, We currently segregate ceph pool PG allocation using the crush device class ruleset as described: https://ceph.com/community/new-luminous-crush-device-classes/ simply using the following command to define the rule : ceph osd crush rule create-replicated <RULE> default host <DEVICE CLASS>
However, we noticed that the rule is not strict in certain scenarios. By that, I mean that if there is no OSD of the specific device class ceph will allocate PG for this pool to any other OSD available ( creating an issue with the PG calculation when we want to add new pool) Simple scenario : 1. create 1 Pool : <pool1> , replication 2 with 4 nodes , 1 OSD each . belonging to class <pool1> 2. remove all OSD ( delete them ) 3. create 4 new OSD (using same disk but different ID) but this time tag them with class <pool2> 4. Try to create pool <pool2> -> this will fail with the pool creation will fail with output : Error ERANGE: pg_num 256 size 2 would mean 1024 total pgs, which exceeds max 800 (mon_max_pg_per_osd 200 * num_in_osds 4)" Pool1 simply started allocating PG to OSD that doesn't belong to the ruleset Which leads me to the following question: is there a way to make the crush rule a hard requirement. E.g : if we do not have any osd matching the device class , it won't start trying to allocate pg to OSD that doesn't match it? Is there any way to prevent pool 1 to use the OSD ? -- Dr. Benoit Hudzia Mobile (UK): +44 (0) 75 346 78673 Mobile (IE): +353 (0) 89 219 3675 Email: ben...@stratoscale.com Web <http://www.stratoscale.com/> | Blog <http://www.stratoscale.com/blog/> | Twitter <https://twitter.com/Stratoscale> | Google+ <https://plus.google.com/u/1/b/108421603458396133912/108421603458396133912/posts> | Linkedin <https://www.linkedin.com/company/stratoscale>
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
