To be more specific, here is the detail of the producer latency after and during the move on the test cluster
50 records sent, 10.0 records/sec (0.01 MB/sec), 1.8 ms avg latency, 3.0 ms max latency. 50 records sent, 10.0 records/sec (0.01 MB/sec), 2.1 ms avg latency, 8.0 ms max latency. 50 records sent, 10.0 records/sec (0.01 MB/sec), 1.8 ms avg latency, 3.0 ms max latency. 50 records sent, 10.0 records/sec (0.01 MB/sec), 2.1 ms avg latency, 10.0 ms max latency. 50 records sent, 10.0 records/sec (0.01 MB/sec), 1.7 ms avg latency, 2.0 ms max latency. 50 records sent, 10.0 records/sec (0.01 MB/sec), 278.9 ms avg latency, 1756.0 ms max latency. 51 records sent, 10.2 records/sec (0.01 MB/sec), 5.3 ms avg latency, 28.0 ms max latency. 50 records sent, 10.0 records/sec (0.01 MB/sec), 8.0 ms avg latency, 81.0 ms max latency. 50 records sent, 10.0 records/sec (0.01 MB/sec), 6.6 ms avg latency, 56.0 ms max latency. 50 records sent, 10.0 records/sec (0.01 MB/sec), 8.2 ms avg latency, 90.0 ms max latency. 51 records sent, 10.0 records/sec (0.01 MB/sec), 8.4 ms avg latency, 78.0 ms max latency. 51 records sent, 10.0 records/sec (0.01 MB/sec), 5.9 ms avg latency, 30.0 ms max latency. 50 records sent, 9.9 records/sec (0.01 MB/sec), 5.5 ms avg latency, 33.0 ms max latency. 51 records sent, 10.0 records/sec (0.01 MB/sec), 12.2 ms avg latency, 263.0 ms max latency. 47 records sent, 9.2 records/sec (0.01 MB/sec), 169.4 ms avg latency, 1173.0 ms max latency. 54 records sent, 10.7 records/sec (0.01 MB/sec), 82.3 ms avg latency, 739.0 ms max latency. 51 records sent, 10.0 records/sec (0.01 MB/sec), 4.7 ms avg latency, 52.0 ms max latency. Le mer. 13 avr. 2022 à 08:54, Fares Oueslati <oueslati.fa...@gmail.com> a écrit : > Hi Luke, > > For now I’m validating the operation on a 3 brokers test cluster with > ~50Gb of data on a single topic (generated using > kafka-producer-perf-test.sh) with one test producer that is reporting a > spike of latency during the operation, going from 2ms on average to 2 > seconds. I’m using ssd disks on gcp fyi. > > The real clusters where I need to move all the partitions have got ~2Tb of > data with several active clients. > > I’m considering throttling « manually » by moving a small set of topics > at a time and separating large topics. > > Wdyt! > > Thank you > > Le mer. 13 avr. 2022 à 04:56, Luke Chen <show...@gmail.com> a écrit : > >> Hi Fares, >> >> > Are you aware of any way to throttle the movement of data between disks? >> >> Interesting question! We've never considered the throttle in disk IO. >> Does it impact the normal throughput a lot? >> >> Thank you. >> Luke >> >> On Wed, Apr 13, 2022 at 1:50 AM Fares Oueslati <oueslati.fa...@gmail.com> >> wrote: >> >> > Thanks Luke for your answer. >> > >> > Are you aware of any way to throttle the movement of data between disks? >> > The —throttle option of the ˋKafka-reassign-partitions` allows to >> throttle >> > inter broker throughput only. >> > >> > >> > Le jeu. 7 avr. 2022 à 05:11, Luke Chen <show...@gmail.com> a écrit : >> > >> > > Hi Fares, >> > > >> > > I don't know if there is other simpler solution, but I think the >> > > `kafka-reassign-partitions` command is the safest way. >> > > >> > > >> > > Thank you. >> > > Luke >> > > >> > > On Wed, Apr 6, 2022 at 11:32 PM Fares Oueslati < >> oueslati.fa...@gmail.com >> > > >> > > wrote: >> > > >> > > > Hey 👋 >> > > > I am using a jbod setup in a 2.8 Kafka cluster. >> > > > >> > > > I started with only one disk in my JBOD, all partitions are on one >> > volume >> > > > (one log.dir) >> > > > >> > > > I have added a disk with the right log.dir and the brokers are well >> > > > configured, I would like to move all replicas of all partitions >> without >> > > > exception from the first volume to the new one. >> > > > >> > > > With the `kafka-reassign-partitions` command it seems to be a bit >> too >> > > much >> > > > trouble. >> > > > I need to generate a `Proposed partition reassignment configuration` >> > and >> > > > then modify the paths to the log_dirs dynamically according to what >> is >> > in >> > > > the `replicas` list. >> > > > >> > > > It can be automated but I wonder if there is a simpler solution for >> my >> > > > relatively simple need. >> > > > >> > > > Thanks >> > > > >> > > >> > >> >
