Thanks Peter for running this experiment. That looks sorta normal. It looks like Connect is deciding to use 10 total tasks and doesn't care which ones do what. Ideally you'd see the MirrorSourceConnector tasks evenly divided, since they do the bulk of the work -- but that doesn't seem to be the case with your selection of parameters.
I'd recommend bumping up the tasks.max a lot higher than 4 in order to achieve finer-grained workloads and a more even balance. Ryanne On Mon, Mar 23, 2020 at 9:58 AM Péter Sinóros-Szabó <peter.sinoros-sz...@transferwise.com.invalid> wrote: > so I made some tests with tasks.max = 4 > > with 2 instances: > - instance 1: 4 MirrorSourceConnector, 1 MirrorHeartbeatConnector tasks > - instance 2: 4 MirrorCheckpointConnector, 1 MirrorHeartbeatConnector tasks > > with 3 instances: > - instance 1: 3 MirrorCheckpointConnector tasks > - instance 2: 3 MirrorSourceConnector tasks, 1 MirrorHeartbeatConnector > task > - instance 3: 1 MirrorSourceConnector, 1 MirrorCheckpointConnector task, > 1 MirrorHeartbeatConnector task > > with 4 instances: > - instance 1: 3 MirrorCheckpointConnector tasks > - instance 2: 2 MirrorSourceConnector tasks, 1 MirrorHeartbeatConnector > task > - instance 3: 1 MirrorSourceConnector task, 1 MirrorCheckpointConnector > task > - instance 4: 1 MirrorSourceConnector task, 1 MirrorHeartbeatConnector task > > So it seems that it is not well balanced, but can be scaled somewhat, not > ideal. > Is this how it should work? > > Peter > > On Fri, 20 Mar 2020 at 20:58, Ryanne Dolan <ryannedo...@gmail.com> wrote: > > > Peter, what happens when you add an additional node? Usually Connect will > > detect it and rebalance tasks accordingly. I'm wondering if that > mechanism > > isn't working for you. > > > > Ryanne > > > > On Fri, Mar 20, 2020 at 2:40 PM Péter Sinóros-Szabó > > <peter.sinoros-sz...@transferwise.com.invalid> wrote: > > > > > Well, I don't know much about herders. If you can give some idea how to > > > check it, I will try. > > > > > > Peter > > > > > > On Fri, 20 Mar 2020 at 17:47, Ryanne Dolan <ryannedo...@gmail.com> > > wrote: > > > > > > > Hmm, that's weird. I'd expect the type of tasks to be evenly > > distributed > > > as > > > > well. Is it possible one of the internal topics are misconfigured > s.t. > > > the > > > > Herders aren't functioning correctly? > > > > > > > > Ryanne > > > > > > > > On Fri, Mar 20, 2020 at 11:17 AM Péter Sinóros-Szabó > > > > <peter.sinoros-sz...@transferwise.com.invalid> wrote: > > > > > > > > > I use tasks.max = 4. > > > > > > > > > > I see 4 tasks of MirrorSourceConnectors on MM2 instances A. > > > > > I see 4 tasks of MirrorCheckpointConnector and 1 task of > > > > > MirrorHeartbeatConnector on MM2 instance B. > > > > > > > > > > The number of tasks are well distributed, but the type of tasks are > > > not. > > > > > According to Connect documentation I expected 1-3 or 2-2 tasks of > > > > > the MirrorSourceConnectors on the two MM2 instances. > > > > > > > > > > So is this a bug or an expected behaviour? > > > > > > > > > > Thanks, > > > > > Peter > > > > > > > > > > On Fri, 20 Mar 2020 at 15:26, Ryanne Dolan <ryannedo...@gmail.com> > > > > wrote: > > > > > > > > > > > Peter, in Connect the Connectors are only run on the leader node. > > > Most > > > > of > > > > > > the work is done in the Tasks, which should be divided across > > nodes. > > > > Make > > > > > > sure you have tasks.max set to something higher than the default > of > > > 1. > > > > > > > > > > > > Ryanne > > > > > > > > > > > > On Fri, Mar 20, 2020, 8:53 AM Péter Sinóros-Szabó > > > > > > <peter.sinoros-sz...@transferwise.com.invalid> wrote: > > > > > > > > > > > > > Hey, > > > > > > > > > > > > > > I am using MM2 to mirror A cluster to B with tasks.max = 4. > > > > > > > > > > > > > > I started two instances of MM2 and noticed that all > > > > > > MirrorSourceConnectors > > > > > > > were running in one instance and the rest of the connectors in > > the > > > > > other. > > > > > > > > > > > > > > This results in a very uneven resource utilization and also it > > did > > > > not > > > > > > > really spread the mirroring oad between the two nodes. > > > > > > > > > > > > > > I assumed that MM2 will run 2-2 of those connectors in each > > > instance. > > > > > > > > > > > > > > Is this current behaviour as expected or did I miss something > on > > > how > > > > to > > > > > > > configure it better? > > > > > > > > > > > > > > Thanks, > > > > > > > Peter > > > > > > > > > > > > > > > > > > > > > > > > > > > >
