Very low bytesout for internal repartition topic, is it related to RocksDB?

2019-10-13 Thread Xiyuan Hu
Hi, I'm running a Kafka Streams app with windowing function. I noticed that internal topic -repartition has huge lag while the system CPU usage is low and app is stable(join rate is almost 0). The repartition topic is an internal topic and created by the application automatically. The bytes in pe

Re: Running a stand alone Mirror Maker 2.0 cluster and Issues

2019-10-13 Thread Ryanne Dolan
Vishal, the first issue is easy: you must set tasks.max to something above 1 (the default) in order to achieve any parallelism. This property is passed along to the internal Connect workers. It's unfortunate that Connect is not smart enough to default this property to the number of workers. I suspe

Running a stand alone Mirror Maker 2.0 cluster and Issues

2019-10-13 Thread Vishal Santoshi
Using https://github.com/apache/kafka/tree/trunk/connect/mirror as a guide, I have build from source the origin/KIP-382 of https://github.com/apache/kafka.git. I am seeing 2 issues * I brought up 2 processes on 2 different nodes ( they are actually pods on k8s but that should not matter ). They s