Hi Luke, For now I’m validating the operation on a 3 brokers test cluster with ~50Gb of data on a single topic (generated using kafka-producer-perf-test.sh) with one test producer that is reporting a spike of latency during the operation, going from 2ms on average to 2 seconds. I’m using ssd disks on gcp fyi.
The real clusters where I need to move all the partitions have got ~2Tb of data with several active clients. I’m considering throttling « manually » by moving a small set of topics at a time and separating large topics. Wdyt! Thank you Le mer. 13 avr. 2022 à 04:56, Luke Chen <show...@gmail.com> a écrit : > Hi Fares, > > > Are you aware of any way to throttle the movement of data between disks? > > Interesting question! We've never considered the throttle in disk IO. > Does it impact the normal throughput a lot? > > Thank you. > Luke > > On Wed, Apr 13, 2022 at 1:50 AM Fares Oueslati <oueslati.fa...@gmail.com> > wrote: > > > Thanks Luke for your answer. > > > > Are you aware of any way to throttle the movement of data between disks? > > The —throttle option of the ˋKafka-reassign-partitions` allows to > throttle > > inter broker throughput only. > > > > > > Le jeu. 7 avr. 2022 à 05:11, Luke Chen <show...@gmail.com> a écrit : > > > > > Hi Fares, > > > > > > I don't know if there is other simpler solution, but I think the > > > `kafka-reassign-partitions` command is the safest way. > > > > > > > > > Thank you. > > > Luke > > > > > > On Wed, Apr 6, 2022 at 11:32 PM Fares Oueslati < > oueslati.fa...@gmail.com > > > > > > wrote: > > > > > > > Hey 👋 > > > > I am using a jbod setup in a 2.8 Kafka cluster. > > > > > > > > I started with only one disk in my JBOD, all partitions are on one > > volume > > > > (one log.dir) > > > > > > > > I have added a disk with the right log.dir and the brokers are well > > > > configured, I would like to move all replicas of all partitions > without > > > > exception from the first volume to the new one. > > > > > > > > With the `kafka-reassign-partitions` command it seems to be a bit too > > > much > > > > trouble. > > > > I need to generate a `Proposed partition reassignment configuration` > > and > > > > then modify the paths to the log_dirs dynamically according to what > is > > in > > > > the `replicas` list. > > > > > > > > It can be automated but I wonder if there is a simpler solution for > my > > > > relatively simple need. > > > > > > > > Thanks > > > > > > > > > >
