Thanks a lot Ryanne. That was really very helpful.
On Thu, Aug 20, 2020, 11:49 PM Ryanne Dolan wrote:
> > Can we configure tasks.max for each of these connectors separately?
>
> I don't believe that's currently possible. If you need fine-grained control
> over each Connector like that, you might
> Can we configure tasks.max for each of these connectors separately?
I don't believe that's currently possible. If you need fine-grained control
over each Connector like that, you might consider running MM2's Connectors
manually on a bunch of Connect clusters. This requires more effort to set
up,
Thanks, Ryanne. That answers my questions. I was actually missing this
"tasks.max" property. Thanks for pointing that out.
Furthermore, as per the KIP of Mirror Maker 2.0, there are 3 types of
connectors in a Mirror Maker Cluster:
1. KafkaSourceConnector - focus on replicating topic partitions
Ananya, see responses below.
> Can this number of workers be configured?
The number of workers is not exactly configurable, but you can control it
by spinning up drivers and using the '--clusters' flag. A driver instance
without '--clusters' will run one worker for each A->B replication flow. So
Any help here would be greatly appreciated.
On Sat, Aug 8, 2020, 12:13 PM Ananya Sen wrote:
> Thank you Ryanne for the quick response.
> I further want to clarify a few points.
>
> The mirror maker 2.0 is based on the Kafka Connect framework. In Kafka
> connect we have multiple workers and each
Thank you Ryanne for the quick response.
I further want to clarify a few points.
The mirror maker 2.0 is based on the Kafka Connect framework. In Kafka connect
we have multiple workers and each worker has some assigned task. To map this to
Mirror Maker 2.0, A mirror Maker will driver have some
Ananya, yes the driver is distributed, but each worker only communicates
via kafka. They do not listen on any ports.
Ryanne
On Sat, Jul 11, 2020, 11:28 AM Ananya Sen wrote:
> Hi
>
> I was exploring the Mirror maker 2.0. I read through this
>
> https://cwiki.apache.org/confluence/display/KAFKA/K
Hi
I was exploring the Mirror maker 2.0. I read through this
https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0
documentation
and I have a few questions.
1. For running mirror maker as a dedicated mirror maker cluster, the
documentation specifies a config file and
