This works better than my solution. It allows the cluster to put more PGs on
the systems with more space on them:
# for pg in $(ceph pg ls-by-pool cephfs_data_ec62 -f json | jq -r
'.pg_stats[].pgid'); do
> echo $pg
> for osd in $(ceph pg map $pg -f json | jq -r '.up[]'); do
> ceph osd find $osd | jq -r '.host'
> done | sort | uniq -c | sort -n -k1
> done
8.0
1 excalibur
1 mandalaybay
2 aladdin
2 harrahs
2 paris
8.1
1 aladdin
1 excalibur
1 harrahs
1 mirage
2 mandalaybay
2 paris
8.2
1 aladdin
1 mandalaybay
2 harrahs
2 mirage
2 paris
...
Thanks!
Bryan
> On May 13, 2021, at 2:58 AM, Ján Senko <ja...@protonmail.ch> wrote:
>
> Caution: This email is from an external sender. Please do not click links or
> open attachments unless you recognize the sender and know the content is
> safe. Forward suspicious emails to isitbad@.
>
>
>
> Would something like this work?
>
> step take default
> step choose indep 4 type host
> step chooseleaf indep 1 type osd
> step emit
> step take default
> step choose indep 0 type host
> step chooseleaf indep 1 type osd
> step emit
>
> J.
>
> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
>
> On Wednesday, May 12th, 2021 at 17:58, Bryan Stillwell
> <bstillw...@godaddy.com> wrote:
>
>> I'm trying to figure out a CRUSH rule that will spread data out across my
>> cluster as much as possible, but not more than 2 chunks per host.
>>
>> If I use the default rule with an osd failure domain like this:
>>
>> step take default
>>
>> step choose indep 0 type osd
>>
>> step emit
>>
>> I get clustering of 3-4 chunks on some of the hosts:
>>
>> for pg in $(ceph pg ls-by-pool cephfs_data_ec62 -f json | jq -r
>> '.pg_stats[].pgid'); do
>> =======================================================================================
>>
>>> echo $pg
>>>
>>> for osd in $(ceph pg map $pg -f json | jq -r '.up[]'); do
>>>
>>> ceph osd find $osd | jq -r '.host'
>>>
>>> done | sort | uniq -c | sort -n -k1
>>
>> 8.0
>>
>> 1 harrahs
>>
>> 3 paris
>>
>> 4 aladdin
>>
>> 8.1
>>
>> 1 aladdin
>>
>> 1 excalibur
>>
>> 2 mandalaybay
>>
>> 4 paris
>>
>> 8.2
>>
>> 1 harrahs
>>
>> 2 aladdin
>>
>> 2 mirage
>>
>> 3 paris
>>
>> ...
>>
>> However, if I change the rule to use:
>>
>> step take default
>>
>> step choose indep 0 type host
>>
>> step chooseleaf indep 2 type osd
>>
>> step emit
>>
>> I get the data spread across 4 hosts with 2 chunks per host:
>>
>> for pg in $(ceph pg ls-by-pool cephfs_data_ec62 -f json | jq -r
>> '.pg_stats[].pgid'); do
>> =======================================================================================
>>
>>> echo $pg
>>>
>>> for osd in $(ceph pg map $pg -f json | jq -r '.up[]'); do
>>>
>>> ceph osd find $osd | jq -r '.host'
>>>
>>> done | sort | uniq -c | sort -n -k1
>>>
>>> done
>>
>> 8.0
>>
>> 2 aladdin
>>
>> 2 harrahs
>>
>> 2 mandalaybay
>>
>> 2 paris
>>
>> 8.1
>>
>> 2 aladdin
>>
>> 2 harrahs
>>
>> 2 mandalaybay
>>
>> 2 paris
>>
>> 8.2
>>
>> 2 harrahs
>>
>> 2 mandalaybay
>>
>> 2 mirage
>>
>> 2 paris
>>
>> ...
>>
>> Is it possible to get the data to spread out over more hosts? I plan on
>> expanding the cluster in the near future and would like to see more hosts
>> get 1 chunk instead of 2.
>>
>> Also, before you recommend adding two more hosts and switching to a
>> host-based failure domain, the cluster is on a variety of hardware with
>> between 2-6 drives per host and drives that are 4TB-12TB in size (it's part
>> of my home lab).
>>
>> Thanks,
>>
>> Bryan
>>
>> ceph-users mailing list -- ceph-users@ceph.io
>>
>> To unsubscribe send an email to ceph-users-le...@ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
