Ok, I understand. But let's say, I am foolish and I run MM2 at the source. How can I notice if there is an issue because of that?
I think I will get some kind of TimeoutExceptions, am I right? Will MM2 actually drop messages in that case? Peter On Thu, 9 Jan 2020 at 21:16, Ryanne Dolan <ryannedo...@gmail.com> wrote: > +1, it is preferable to run MM2 at the target/destination cluster. > Basically, producer lag is more problematic than consumer lag, so you want > the producer as close to the target cluster as possible. > > Also take a look at the "--clusters" option of the connect-mirror-maker.sh > command, which lets you specify which clusters are nearby each MM2 node. If > specified, the node will only produce to those specific clusters (and > consume from everywhere else). > > Ryanne > > On Thu, Jan 9, 2020 at 10:32 AM Andrew Otto <o...@wikimedia.org> wrote: > > > Hi Peter, > > > > My understanding here comes from MirrorMaker 1, but I believe it holds > for > > MM2 (someone correct me if I am wrong!) > > For the most part, if you have no latency or connectivity issues, running > > MM at the source will be fine. However, the failure scenario is > different > > if something goes wrong. > > > > When running at the destination, it is the kafka consumer that has to > cross > > the network boundary. If the consumer can't consume, it can always pick > > off from where it left off later. > > > > When running at the source, it is the kafka producer that has to cross > the > > network boundary. If the producer can't produce, it will eventually drop > > messages. > > > > > > > > On Thu, Jan 9, 2020 at 11:28 AM Péter Sinóros-Szabó > > <peter.sinoros-sz...@transferwise.com.invalid> wrote: > > > > > Hey, > > > > > > I am thinking about where (well in which AWS region) should I run MM2. > > > I might be wrong, but as I know it is better to run it close to the > > > destination cluster. > > > But for other reasons, it would be much easier for me to run it at the > > > source. > > > So is it still advised to run MM2 at the destination? > > > Latency between source and destination is about 32ms. > > > What are the downsides if I run it at the source? > > > > > > Thanks, > > > Peter > > > > > >
