Peter, what happens when you add an additional node? Usually Connect will detect it and rebalance tasks accordingly. I'm wondering if that mechanism isn't working for you.
Ryanne On Fri, Mar 20, 2020 at 2:40 PM Péter Sinóros-Szabó <peter.sinoros-sz...@transferwise.com.invalid> wrote: > Well, I don't know much about herders. If you can give some idea how to > check it, I will try. > > Peter > > On Fri, 20 Mar 2020 at 17:47, Ryanne Dolan <ryannedo...@gmail.com> wrote: > > > Hmm, that's weird. I'd expect the type of tasks to be evenly distributed > as > > well. Is it possible one of the internal topics are misconfigured s.t. > the > > Herders aren't functioning correctly? > > > > Ryanne > > > > On Fri, Mar 20, 2020 at 11:17 AM Péter Sinóros-Szabó > > <peter.sinoros-sz...@transferwise.com.invalid> wrote: > > > > > I use tasks.max = 4. > > > > > > I see 4 tasks of MirrorSourceConnectors on MM2 instances A. > > > I see 4 tasks of MirrorCheckpointConnector and 1 task of > > > MirrorHeartbeatConnector on MM2 instance B. > > > > > > The number of tasks are well distributed, but the type of tasks are > not. > > > According to Connect documentation I expected 1-3 or 2-2 tasks of > > > the MirrorSourceConnectors on the two MM2 instances. > > > > > > So is this a bug or an expected behaviour? > > > > > > Thanks, > > > Peter > > > > > > On Fri, 20 Mar 2020 at 15:26, Ryanne Dolan <ryannedo...@gmail.com> > > wrote: > > > > > > > Peter, in Connect the Connectors are only run on the leader node. > Most > > of > > > > the work is done in the Tasks, which should be divided across nodes. > > Make > > > > sure you have tasks.max set to something higher than the default of > 1. > > > > > > > > Ryanne > > > > > > > > On Fri, Mar 20, 2020, 8:53 AM Péter Sinóros-Szabó > > > > <peter.sinoros-sz...@transferwise.com.invalid> wrote: > > > > > > > > > Hey, > > > > > > > > > > I am using MM2 to mirror A cluster to B with tasks.max = 4. > > > > > > > > > > I started two instances of MM2 and noticed that all > > > > MirrorSourceConnectors > > > > > were running in one instance and the rest of the connectors in the > > > other. > > > > > > > > > > This results in a very uneven resource utilization and also it did > > not > > > > > really spread the mirroring oad between the two nodes. > > > > > > > > > > I assumed that MM2 will run 2-2 of those connectors in each > instance. > > > > > > > > > > Is this current behaviour as expected or did I miss something on > how > > to > > > > > configure it better? > > > > > > > > > > Thanks, > > > > > Peter > > > > > > > > > > > > > > >
