I bumped tasks.max to 10, but it does not help if I only have two
instances: all MirrorSourceConnector tasks are still in one instance.
But if I start more instances in case I need more bandwidth, it will help
to spread the load, so this is ok for me so far.
Also KAFKA-9352 will help to get better
Hi Peter,
Not sure this is what you're looking for -
https://issues.apache.org/jira/browse/KAFKA-9352
On Mon, Mar 23, 2020 at 11:37 PM Ryanne Dolan wrote:
> Thanks Peter for running this experiment. That looks sorta normal. It looks
> like Connect is deciding to use 10 total tasks and doesn't c
Thanks Peter for running this experiment. That looks sorta normal. It looks
like Connect is deciding to use 10 total tasks and doesn't care which ones
do what. Ideally you'd see the MirrorSourceConnector tasks evenly divided,
since they do the bulk of the work -- but that doesn't seem to be the cas
so I made some tests with tasks.max = 4
with 2 instances:
- instance 1: 4 MirrorSourceConnector, 1 MirrorHeartbeatConnector tasks
- instance 2: 4 MirrorCheckpointConnector, 1 MirrorHeartbeatConnector tasks
with 3 instances:
- instance 1: 3 MirrorCheckpointConnector tasks
- instance 2: 3 MirrorSou
Peter, what happens when you add an additional node? Usually Connect will
detect it and rebalance tasks accordingly. I'm wondering if that mechanism
isn't working for you.
Ryanne
On Fri, Mar 20, 2020 at 2:40 PM Péter Sinóros-Szabó
wrote:
> Well, I don't know much about herders. If you can give
Well, I don't know much about herders. If you can give some idea how to
check it, I will try.
Peter
On Fri, 20 Mar 2020 at 17:47, Ryanne Dolan wrote:
> Hmm, that's weird. I'd expect the type of tasks to be evenly distributed as
> well. Is it possible one of the internal topics are misconfigured
Hmm, that's weird. I'd expect the type of tasks to be evenly distributed as
well. Is it possible one of the internal topics are misconfigured s.t. the
Herders aren't functioning correctly?
Ryanne
On Fri, Mar 20, 2020 at 11:17 AM Péter Sinóros-Szabó
wrote:
> I use tasks.max = 4.
>
> I see 4 task
I use tasks.max = 4.
I see 4 tasks of MirrorSourceConnectors on MM2 instances A.
I see 4 tasks of MirrorCheckpointConnector and 1 task of
MirrorHeartbeatConnector on MM2 instance B.
The number of tasks are well distributed, but the type of tasks are not.
According to Connect documentation I expec
Peter, in Connect the Connectors are only run on the leader node. Most of
the work is done in the Tasks, which should be divided across nodes. Make
sure you have tasks.max set to something higher than the default of 1.
Ryanne
On Fri, Mar 20, 2020, 8:53 AM Péter Sinóros-Szabó
wrote:
> Hey,
>
> I
Hey,
I am using MM2 to mirror A cluster to B with tasks.max = 4.
I started two instances of MM2 and noticed that all MirrorSourceConnectors
were running in one instance and the rest of the connectors in the other.
This results in a very uneven resource utilization and also it did not
really spre
10 matches
Mail list logo
