Michal; Is the balancer (upmap balancer) running (ceph balancer status) and undoing some of the upmaps the pgremapper is introducing?
Respectfully, *Wes Dillingham* LinkedIn <http://www.linkedin.com/in/wesleydillingham> w...@wesdillingham.com On Sat, Jun 21, 2025 at 2:05 AM Michal Strnad <michal.str...@cesnet.cz> wrote: > Hi. > > Over the past few days, we've been working on migrating data from > machine A to machine B using the pgremapper tool, but we haven’t been > able to achieve the expected results. > > As part of our testing, we set up a small Ceph cluster with several > monitors, managers, and servers with OSDs.We applied the flags noout, > nobackfill, norecovery, and norebalance, and then added additional > servers with OSDs. While Ceph did allocate PG replicas to the newly > added OSDs, the actual data didn’t move due to the active flags. We then > attempted to use pgremapper to migrate all PGs from one server to the > new one, removing or negating the flags in the process. However, we > frequently failed to complete the migration of all data/PGs. > Are we overlooking something? Does anyone have a reliable, step-by-step > procedure we can follow to perform this correctly? > > Any help would be greatly appreciated! > > Michal > > > On 3/19/25 08:13, Janne Johansson wrote: > >> The safest approach would be to use the upmap-remapped.py tool > developed by Dan at CERN. See [1] for details. > >> > >> The idea is to leverage the upmap load balancer to progressively > migrate the data to the new servers, minimizing performance impact on the > cluster and clients. I like to create the OSDs ahead of time on the nodes > that I initially place in a root directory called ‘closet’. > >> > >> I then apply the norebalance flag (ceph osd set norebalance), disable > the balancer (ceph balancer off), move the new nodes with already > provisioned OSDs to their final location (rack), run ./upmap-remapped.py to > bring all PGs back to active+clean state, remove the norebalance flag (ceph > osd unset norebalance), re-enable the balancer (ceph balancer on) and watch > data moving progressively as the upmap balancer executes its plans. > > > > We do exactly that also, sometimes using pgremapper instead of > > upmap-remapper.py, but the effect is the same. Make the changes with > > norebalance, upmap the PGs to be happy where they are until we unset > > norebalance and let the ceph balancer correct it X% at a time. > > > > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io > _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
