Vishal, the first issue is easy: you must set tasks.max to something above 1 (the default) in order to achieve any parallelism. This property is passed along to the internal Connect workers. It's unfortunate that Connect is not smart enough to default this property to the number of workers. I suspect that will improve before long.
For the second issue, is it possible you are missing the offsets topic? It should exist alongside the config and status topics. Connect should create this topic, but there are various reasons this can fail, e.g. if the replication factor is misconfigured. You can try creating this topic manually or changing offsets.storage.replication.factor. Ryanne On Sun, Oct 13, 2019, 5:13 PM Vishal Santoshi <vishal.santo...@gmail.com> wrote: > Using https://github.com/apache/kafka/tree/trunk/connect/mirror as a > guide, > I have build from source the origin/KIP-382 of > https://github.com/apache/kafka.git. > > I am seeing 2 issues > > * I brought up 2 processes on 2 different nodes ( they are actually pods on > k8s but that should not matter ). They share the mm2.properties file and > are replicating ( 1-way ) 3 topics with 8 partitions in total. That seems > to be the way to create a standalone mm2 cluster. I do not however see( at > least the mbeans do not show ) any attempt to rebalance. > > https://github.com/apache/kafka/tree/trunk/connect/mirror#monitoring-an-mm2-process > mbeans > are all on a single node > > > > * I restart the processes on the 2 nodes ( hard stop ans start ). The > offsets for replication seem to be reset to the earliest, as if it is a > brand new mirroring. It is also obvious from the > "record-age-ms-avg|replication-latency-ms-avg" > which I track through the restart. > > > > > This implies that > > > 1. Load balancing by rebalancing is not working. I cannot scale up or down > by adding nodes to the mm2 cluster or removing them. > > 2. Restore on a mirror is not working. If the MM2 cluster is brought down, > it does not start mirroring from the last known state. I see the, > state/config topics etc created as expected.. > > > > > > The mm2.properties is pretty mimimal > > > *clusters = a , b* > > > > *a.bootstrap.servers = k.....* > > *b.bootstrap.servers = k.....* > > > *# only allow replication dr1 -> dr2* > > *a->b.enabled = true* > > *a->b.topics = act_search_page* > > *a->b.emit.heartbeats.enabled = false* > > > *b->a..enabled = false* > > *b->a.emit.heartbeats.enabled = false* > > > > > > > What do you think is the issue ? > > > Thanks >
