Hi Fares, Thanks for sharing the information.
> I’m considering throttling « manually » by moving a small set of topics at a time and separating large topics. Yes, this is also the option I can think of. Not sure if there are other suggestions from the community. Thank you. Luke On Wed, Apr 13, 2022 at 4:09 PM Fares Oueslati <oueslati.fa...@gmail.com> wrote: > To be more specific, here is the detail of the producer latency after and > during the move on the test cluster > > 50 records sent, 10.0 records/sec (0.01 MB/sec), 1.8 ms avg latency, 3.0 ms > max latency. > 50 records sent, 10.0 records/sec (0.01 MB/sec), 2.1 ms avg latency, 8.0 ms > max latency. > 50 records sent, 10.0 records/sec (0.01 MB/sec), 1.8 ms avg latency, 3.0 ms > max latency. > 50 records sent, 10.0 records/sec (0.01 MB/sec), 2.1 ms avg latency, 10.0 > ms max latency. > 50 records sent, 10.0 records/sec (0.01 MB/sec), 1.7 ms avg latency, 2.0 ms > max latency. > 50 records sent, 10.0 records/sec (0.01 MB/sec), 278.9 ms avg latency, > 1756.0 ms max latency. > 51 records sent, 10.2 records/sec (0.01 MB/sec), 5.3 ms avg latency, 28.0 > ms max latency. > 50 records sent, 10.0 records/sec (0.01 MB/sec), 8.0 ms avg latency, 81.0 > ms max latency. > 50 records sent, 10.0 records/sec (0.01 MB/sec), 6.6 ms avg latency, 56.0 > ms max latency. > 50 records sent, 10.0 records/sec (0.01 MB/sec), 8.2 ms avg latency, 90.0 > ms max latency. > 51 records sent, 10.0 records/sec (0.01 MB/sec), 8.4 ms avg latency, 78.0 > ms max latency. > 51 records sent, 10.0 records/sec (0.01 MB/sec), 5.9 ms avg latency, 30.0 > ms max latency. > 50 records sent, 9.9 records/sec (0.01 MB/sec), 5.5 ms avg latency, 33.0 ms > max latency. > 51 records sent, 10.0 records/sec (0.01 MB/sec), 12.2 ms avg latency, 263.0 > ms max latency. > 47 records sent, 9.2 records/sec (0.01 MB/sec), 169.4 ms avg latency, > 1173.0 ms max latency. > 54 records sent, 10.7 records/sec (0.01 MB/sec), 82.3 ms avg latency, 739.0 > ms max latency. > 51 records sent, 10.0 records/sec (0.01 MB/sec), 4.7 ms avg latency, 52.0 > ms max latency. > > Le mer. 13 avr. 2022 à 08:54, Fares Oueslati <oueslati.fa...@gmail.com> a > écrit : > > > Hi Luke, > > > > For now I’m validating the operation on a 3 brokers test cluster with > > ~50Gb of data on a single topic (generated using > > kafka-producer-perf-test.sh) with one test producer that is reporting a > > spike of latency during the operation, going from 2ms on average to 2 > > seconds. I’m using ssd disks on gcp fyi. > > > > The real clusters where I need to move all the partitions have got ~2Tb > of > > data with several active clients. > > > > I’m considering throttling « manually » by moving a small set of topics > > at a time and separating large topics. > > > > Wdyt! > > > > Thank you > > > > Le mer. 13 avr. 2022 à 04:56, Luke Chen <show...@gmail.com> a écrit : > > > >> Hi Fares, > >> > >> > Are you aware of any way to throttle the movement of data between > disks? > >> > >> Interesting question! We've never considered the throttle in disk IO. > >> Does it impact the normal throughput a lot? > >> > >> Thank you. > >> Luke > >> > >> On Wed, Apr 13, 2022 at 1:50 AM Fares Oueslati < > oueslati.fa...@gmail.com> > >> wrote: > >> > >> > Thanks Luke for your answer. > >> > > >> > Are you aware of any way to throttle the movement of data between > disks? > >> > The —throttle option of the ˋKafka-reassign-partitions` allows to > >> throttle > >> > inter broker throughput only. > >> > > >> > > >> > Le jeu. 7 avr. 2022 à 05:11, Luke Chen <show...@gmail.com> a écrit : > >> > > >> > > Hi Fares, > >> > > > >> > > I don't know if there is other simpler solution, but I think the > >> > > `kafka-reassign-partitions` command is the safest way. > >> > > > >> > > > >> > > Thank you. > >> > > Luke > >> > > > >> > > On Wed, Apr 6, 2022 at 11:32 PM Fares Oueslati < > >> oueslati.fa...@gmail.com > >> > > > >> > > wrote: > >> > > > >> > > > Hey 👋 > >> > > > I am using a jbod setup in a 2.8 Kafka cluster. > >> > > > > >> > > > I started with only one disk in my JBOD, all partitions are on one > >> > volume > >> > > > (one log.dir) > >> > > > > >> > > > I have added a disk with the right log.dir and the brokers are > well > >> > > > configured, I would like to move all replicas of all partitions > >> without > >> > > > exception from the first volume to the new one. > >> > > > > >> > > > With the `kafka-reassign-partitions` command it seems to be a bit > >> too > >> > > much > >> > > > trouble. > >> > > > I need to generate a `Proposed partition reassignment > configuration` > >> > and > >> > > > then modify the paths to the log_dirs dynamically according to > what > >> is > >> > in > >> > > > the `replicas` list. > >> > > > > >> > > > It can be automated but I wonder if there is a simpler solution > for > >> my > >> > > > relatively simple need. > >> > > > > >> > > > Thanks > >> > > > > >> > > > >> > > >> > > >
