[ ed: snag during moderation (somehow a newline was interpolated in the
Subject), so I’m sending this on behalf of kasper_steenga...@hotmail.com
<mailto:kasper_steenga...@hotmail.com> , to whom replies should be sent]
I'm managing a ceph cluster with +1K OSDs distributed accross 56 host.
Until now the crush rule used is the default replicated rule, but I want to
change that in order to implement failure domain on rack level.
Current plan is to
- Disable rebalancing by executing - ceph osd set norebalance;
- Add Rack to crushmap and distribute the hosts accordingly (8 in each.) by
using the built in commands
- ceph osd crush add-bucket rack1 rack root=default
- ceph osd crush move osd-host1 rack=rack1
- Create the new rack split rule with command
- ceph osd crush rule create-replicated rack_split default rack
- Set the rule across all my pools
- for p in $(ceph osd lspools | cut -d' ' -f 2) ; do echo $p $(ceph osd
pool set $p crush_rule rack_split) ; done
- Finally enable rebalancing - ceph osd unset norebalance;
However I'm concerned with the amount of data that needs to be rebalanced, since
the cluster holds multiple PB, and I'm looking for review of/input for my plan,
as well as words of advice/experience from someone who have been in similar
situations.
——
[
ed: You only include commands for one CRUSH `rack` — would you create multiple
`rack` CRUSH buckets, at least three of them?
Are all of your pools replicated? No EC pools for RGW buckets, CephFS data,
etc?
What OSD media and networking does this cluster have? HDDs will be much slower
and much more impacted during the process than SSDs. Is your client workload
24x7? Which Ceph release? These factors inform how impactful the grand shuffle
will be. Are your mon DBs on SSDs?
A popular strategy is to use upmap-remapped.py to freeze all of the PG mappings
before unsetting the norebalance flag, then the balancer will gradually undo
the mappings as it moves data to where it now belongs. This process has
built-in throttling.
]
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
