> On March 4, 2015, midnight, Guozhang Wang wrote: > > core/src/main/scala/kafka/tools/MirrorMaker.scala, line 67 > > <https://reviews.apache.org/r/31706/diff/1/?file=883687#file883687line67> > > > > Are we sure that one producer io thread is sufficient for all cases? > > Jiangjie Qin wrote: > The assumption is that if more producers are needed, just create more > mirror maker instances. > > Guozhang Wang wrote: > I think adding more MM instances is not ideal right? If we have n hosts > for MM but need m producers to keep up the traffic where m > n, we will end > up with more-than-one MM in each host, which is a quite a waste of resources > isn't it?
Oh, by instance I just mean another mirror maker process. Not a mirror maker cluster. > On March 4, 2015, midnight, Guozhang Wang wrote: > > core/src/main/scala/kafka/tools/MirrorMaker.scala, line 682 > > <https://reviews.apache.org/r/31706/diff/1/?file=883687#file883687line682> > > > > Instead of a wrapper rebalancer, I think it is cleaner to just > > instantiate the ConsumerRebalanceListener interface and pass-in the > > producer / mirrormaker object for flush() and commitOffsets(). > > Jiangjie Qin wrote: > I don't quite follow the comment here. Can you elaborate? > > Guozhang Wang wrote: > Instead of > > "class InternalRebalanceListener(Option[ConsumerRebalanceListener]) > extends ConsumerRebalanceListener" > > could we just have > > "class InternalRebalanceListener(KafkaProducer) extends > ConsumerRebalanceListener" > > and in the instantiated callback we can call KafkaProducer.send() and > commitOffsets of MirrorMaker, right? Talked to Guozhang offline. Will keep the code as is because we need to invoke the custom rebalance listener in the internal rebalance listener. - Jiangjie ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31706/#review75071 ----------------------------------------------------------- On March 4, 2015, 12:28 a.m., Jiangjie Qin wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/31706/ > ----------------------------------------------------------- > > (Updated March 4, 2015, 12:28 a.m.) > > > Review request for kafka. > > > Bugs: KAFKA-1997 > https://issues.apache.org/jira/browse/KAFKA-1997 > > > Repository: kafka > > > Description > ------- > > Addressed Guozhang's comments. > > > Diffs > ----- > > > clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java > d5c79e2481d5e9a2524ac2ef6a6879f61cb7cb5f > core/src/main/scala/kafka/consumer/PartitionAssignor.scala > e6ff7683a0df4a7d221e949767e57c34703d5aad > core/src/main/scala/kafka/consumer/ZookeeperConsumerConnector.scala > 5487259751ebe19f137948249aa1fd2637d2deb4 > core/src/main/scala/kafka/javaapi/consumer/ConsumerRebalanceListener.java > 7f45a90ba6676290172b7da54c15ee5dc1a42a2e > core/src/main/scala/kafka/tools/MirrorMaker.scala > 5374280dc97dc8e01e9b3ba61fd036dc13ae48cb > core/src/test/scala/unit/kafka/consumer/PartitionAssignorTest.scala > 543070f4fd3e96f3183cae9ee2ccbe843409ee58 > > core/src/test/scala/unit/kafka/consumer/ZookeeperConsumerConnectorTest.scala > a17e8532c44aadf84b8da3a57bcc797a848b5020 > > Diff: https://reviews.apache.org/r/31706/diff/ > > > Testing > ------- > > > Thanks, > > Jiangjie Qin > >
