I bumped tasks.max to 10, but it does not help if I only have two instances: all MirrorSourceConnector tasks are still in one instance.
But if I start more instances in case I need more bandwidth, it will help to spread the load, so this is ok for me so far. Also KAFKA-9352 will help to get better instance usage, once Connect assigns tasks in all instances :) Peter On Mon, 23 Mar 2020 at 19:07, Ryanne Dolan <ryannedo...@gmail.com> wrote: > Thanks Peter for running this experiment. That looks sorta normal. It looks > like Connect is deciding to use 10 total tasks and doesn't care which ones > do what. Ideally you'd see the MirrorSourceConnector tasks evenly divided, > since they do the bulk of the work -- but that doesn't seem to be the case > with your selection of parameters. > > I'd recommend bumping up the tasks.max a lot higher than 4 in order to > achieve finer-grained workloads and a more even balance. > > Ryanne > > On Mon, Mar 23, 2020 at 9:58 AM Péter Sinóros-Szabó > <peter.sinoros-sz...@transferwise.com.invalid> wrote: > > > so I made some tests with tasks.max = 4 > > > > with 2 instances: > > - instance 1: 4 MirrorSourceConnector, 1 MirrorHeartbeatConnector tasks > > - instance 2: 4 MirrorCheckpointConnector, 1 MirrorHeartbeatConnector > tasks > > > > with 3 instances: > > - instance 1: 3 MirrorCheckpointConnector tasks > > - instance 2: 3 MirrorSourceConnector tasks, 1 MirrorHeartbeatConnector > > task > > - instance 3: 1 MirrorSourceConnector, 1 MirrorCheckpointConnector task, > > 1 MirrorHeartbeatConnector task > > > > with 4 instances: > > - instance 1: 3 MirrorCheckpointConnector tasks > > - instance 2: 2 MirrorSourceConnector tasks, 1 MirrorHeartbeatConnector > > task > > - instance 3: 1 MirrorSourceConnector task, 1 MirrorCheckpointConnector > > task > > - instance 4: 1 MirrorSourceConnector task, 1 MirrorHeartbeatConnector > task > > > > So it seems that it is not well balanced, but can be scaled somewhat, not > > ideal. > > Is this how it should work? > > > > Peter > > > > On Fri, 20 Mar 2020 at 20:58, Ryanne Dolan <ryannedo...@gmail.com> > wrote: > > > > > Peter, what happens when you add an additional node? Usually Connect > will > > > detect it and rebalance tasks accordingly. I'm wondering if that > > mechanism > > > isn't working for you. > > > > > > Ryanne > > > > > > On Fri, Mar 20, 2020 at 2:40 PM Péter Sinóros-Szabó > > > <peter.sinoros-sz...@transferwise.com.invalid> wrote: > > > > > > > Well, I don't know much about herders. If you can give some idea how > to > > > > check it, I will try. > > > > > > > > Peter > > > > > > > > On Fri, 20 Mar 2020 at 17:47, Ryanne Dolan <ryannedo...@gmail.com> > > > wrote: > > > > > > > > > Hmm, that's weird. I'd expect the type of tasks to be evenly > > > distributed > > > > as > > > > > well. Is it possible one of the internal topics are misconfigured > > s.t. > > > > the > > > > > Herders aren't functioning correctly? > > > > > > > > > > Ryanne > > > > > > > > > > On Fri, Mar 20, 2020 at 11:17 AM Péter Sinóros-Szabó > > > > > <peter.sinoros-sz...@transferwise.com.invalid> wrote: > > > > > > > > > > > I use tasks.max = 4. > > > > > > > > > > > > I see 4 tasks of MirrorSourceConnectors on MM2 instances A. > > > > > > I see 4 tasks of MirrorCheckpointConnector and 1 task of > > > > > > MirrorHeartbeatConnector on MM2 instance B. > > > > > > > > > > > > The number of tasks are well distributed, but the type of tasks > are > > > > not. > > > > > > According to Connect documentation I expected 1-3 or 2-2 tasks of > > > > > > the MirrorSourceConnectors on the two MM2 instances. > > > > > > > > > > > > So is this a bug or an expected behaviour? > > > > > > > > > > > > Thanks, > > > > > > Peter > > > > > > > > > > > > On Fri, 20 Mar 2020 at 15:26, Ryanne Dolan < > ryannedo...@gmail.com> > > > > > wrote: > > > > > > > > > > > > > Peter, in Connect the Connectors are only run on the leader > node. > > > > Most > > > > > of > > > > > > > the work is done in the Tasks, which should be divided across > > > nodes. > > > > > Make > > > > > > > sure you have tasks.max set to something higher than the > default > > of > > > > 1. > > > > > > > > > > > > > > Ryanne > > > > > > > > > > > > > > On Fri, Mar 20, 2020, 8:53 AM Péter Sinóros-Szabó > > > > > > > <peter.sinoros-sz...@transferwise.com.invalid> wrote: > > > > > > > > > > > > > > > Hey, > > > > > > > > > > > > > > > > I am using MM2 to mirror A cluster to B with tasks.max = 4. > > > > > > > > > > > > > > > > I started two instances of MM2 and noticed that all > > > > > > > MirrorSourceConnectors > > > > > > > > were running in one instance and the rest of the connectors > in > > > the > > > > > > other. > > > > > > > > > > > > > > > > This results in a very uneven resource utilization and also > it > > > did > > > > > not > > > > > > > > really spread the mirroring oad between the two nodes. > > > > > > > > > > > > > > > > I assumed that MM2 will run 2-2 of those connectors in each > > > > instance. > > > > > > > > > > > > > > > > Is this current behaviour as expected or did I miss something > > on > > > > how > > > > > to > > > > > > > > configure it better? > > > > > > > > > > > > > > > > Thanks, > > > > > > > > Peter > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
