Documentation example correction for per topic configs
The Kafka documentation here ( http://kafka.apache.org/081/documentation.html#topic-config) mentions the following as an example: *> bin/kafka-topics.sh --zookeeper localhost:2181 --create --topic my-topic --partitions 1 --replication-factor 1 --config max.message.bytes=64000 --config flush.messages=1* and *> bin/kafka-topics.sh --zookeeper localhost:2181 --alter --topic my-topic --config max.message.bytes=128000* The configs must be put in double quotes here, i.e. *--config *"*flush.messages=1".* The documentation for 0.8.2 also would need the correction. Regards, Soumyajit Sahu
Kafka Partition Reassignment service (based on an adoption marketplace model)
Hello all, I have a design for a solution to the problem of "partition imbalances in Kafka clusters". It would be great to get some feedback on it. https://soumyajitsahu.wordpress.com/2016/05/11/kafka-partition-reassignment-service-using-an-adoption-marketplace-model/ I have also put a proof-of-concept build at https://github.com/Microsoft/Cluster-Partition-Rebalancer-For-Kafka That might require some changes before being run, as it has some hard-coded methods to fetch my broker and zookeeper list during run-time from my custom file. Thanks, Soumyajit Sahu
High write operations rate on disk
We are running Kafka on AWS EC2 instances (m5.2xlarge) with mounted EBS st1 volume (one on each machine). Occasionally, we have noticed that the write ops/second goes through the roof and we get throttled by AWS while the data throughput wouldn't have changed much. As far as our observation goes, it happens usually after a broker restart. Has anyone else come across this behavior? Thanks!
Re: High write operations rate on disk
Our typical IOPS stays at ~10K write ops/min, but it goes to 37K write ops/min (which is where AWS throttles). The spike in write ops isn't accompanied by any spike in write throughput or produce requests (except for the first few minutes of catch up). The write ops spike stays up (persistently for an hour or two) until we stop the broker ec2 instance for about 30 mins and then start it back. @Liam, no, we are not using log compaction except for a few consumer offset topics and config topic (for Kafka Connect), and schema registry store. @Suman, are you using m5 or r5 instances. Recently, we migrated from r5 to m5, and I wonder if that has a hand in this. We have about 1000 partitions residing on each disk, but I don't think that matters as most of the time the brokers run flawlessly (even during peak traffic hours). Thanks! On Mon, Apr 6, 2020 at 11:39 PM Suman B N wrote: > We too have a similar setup but we never observed any such spikes. > > Are you sure your disk IOPS is good enough? Check if that is throttling. > > After a broker restarts, there might be more traffic as well because of > followers trying to catch up with the leader. > > -Suman > > On Tue, Apr 7, 2020 at 11:59 AM Soumyajit Sahu > wrote: > > > We are running Kafka on AWS EC2 instances (m5.2xlarge) with mounted EBS > st1 > > volume (one on each machine). > > Occasionally, we have noticed that the write ops/second goes through the > > roof and we get throttled by AWS while the data throughput wouldn't have > > changed much. As far as our observation goes, it happens usually after a > > broker restart. > > > > Has anyone else come across this behavior? > > > > Thanks! > > > > > -- > *Suman* > *OlaCabs* >
Re: High write operations rate on disk
@Suman, thanks for confirming. I will dig more then. The instances are dedicated to running Kafka, and so is the mounted volume. @Seva, thanks for the insight. I guess if nothing works, then we will move from st1 to gp2 volumes. On Tue, Apr 7, 2020 at 12:28 AM Suman B N wrote: > We have used st1 volumes and we never saw any issue. > Yes, we are using m-series. Even t-series worked for us :D > > During those spikes, do you observe any background operations going on? > Check server logs, controller logs. > > On Tue, Apr 7, 2020 at 12:49 PM Seva Feldman wrote: > > > ST1 EBS fit only for sequential rights and reads. Once you have many > > partitions on EBS it will be mostly random. > > Interesting to monitor random vs sequential... > > > > We tested kafka on ST1 with 1xx partitions on each EBS and it was > > constantly lagging. > > > > BR > > > > On Tue, Apr 7, 2020 at 10:06 AM Soumyajit Sahu > > > wrote: > > > > > Our typical IOPS stays at ~10K write ops/min, but it goes to 37K write > > > ops/min (which is where AWS throttles). > > > The spike in write ops isn't accompanied by any spike in write > throughput > > > or produce requests (except for the first few minutes of catch up). The > > > write ops spike stays up (persistently for an hour or two) until we > stop > > > the broker ec2 instance for about 30 mins and then start it back. > > > > > > @Liam, no, we are not using log compaction except for a few consumer > > offset > > > topics and config topic (for Kafka Connect), and schema registry store. > > > > > > @Suman, are you using m5 or r5 instances. Recently, we migrated from r5 > > to > > > m5, and I wonder if that has a hand in this. > > > > > > We have about 1000 partitions residing on each disk, but I don't think > > that > > > matters as most of the time the brokers run flawlessly (even during > peak > > > traffic hours). > > > > > > Thanks! > > > > > > On Mon, Apr 6, 2020 at 11:39 PM Suman B N > wrote: > > > > > > > We too have a similar setup but we never observed any such spikes. > > > > > > > > Are you sure your disk IOPS is good enough? Check if that is > > throttling. > > > > > > > > After a broker restarts, there might be more traffic as well because > of > > > > followers trying to catch up with the leader. > > > > > > > > -Suman > > > > > > > > On Tue, Apr 7, 2020 at 11:59 AM Soumyajit Sahu < > > soumyajit.s...@gmail.com > > > > > > > > wrote: > > > > > > > > > We are running Kafka on AWS EC2 instances (m5.2xlarge) with mounted > > EBS > > > > st1 > > > > > volume (one on each machine). > > > > > Occasionally, we have noticed that the write ops/second goes > through > > > the > > > > > roof and we get throttled by AWS while the data throughput wouldn't > > > have > > > > > changed much. As far as our observation goes, it happens usually > > after > > > a > > > > > broker restart. > > > > > > > > > > Has anyone else come across this behavior? > > > > > > > > > > Thanks! > > > > > > > > > > > > > > > > > -- > > > > *Suman* > > > > *OlaCabs* > > > > > > > > > > > > > -- > > Seva Feldman > > VP R&D Mobile Delivery > > [image: ironSource] <http://www.ironsrc.com/> > > > > email sev...@ironsrc.com > > mobile +972544346089 > > > > ironSource HQ - 121 Derech Menachem Begin st. Tel Aviv > > > > > -- > *Suman* > *OlaCabs* >
