Currently it is a manual process. For functional test, I just setup two Kafka clusters locally, mirror between them and keep producing data to one of the cluster. Then try a hard kill / bounce mirror maker to see if messages are lost in target cluster.
Jiangjie (Becket) Qin On 1/21/15, 12:24 PM, "Gwen Shapira" <gshap...@cloudera.com> wrote: >Thanks for the answers. Much clearer now :) > >Unrelated question: How do you test MirrorMaker (especially around data >loss)? >I didn't see any unit-tests or integration tests in trunk. > >Gwen > >On Wed, Jan 21, 2015 at 9:55 AM, Jiangjie Qin <j...@linkedin.com.invalid> >wrote: >> Hi Gwen, >> >> Please see inline answers. I¹ll update them in the KIP as well. >> >> Thanks. >> >> Jiangjie (Becket) Qin >> >> On 1/20/15, 6:39 PM, "Gwen Shapira" <gshap...@cloudera.com> wrote: >> >>>Thanks for the detailed document, Jiangjie. Super helpful. >>> >>>Few questions: >>> >>>1. You mention that "A ConsumerRebalanceListener class is created and >>>could be wired into ZookeeperConsumerConnector to avoid duplicate >>>messages when consumer rebalance occurs in mirror maker." >>> >>>Is this something the user needs to do or configure? or is the wiring >>>of rebalance listener into the zookeeper consumer will be part of the >>>enhancement? >>>In other words, will we need to do anything extra to avoid duplicates >>>during rebalance in MirrorMaker? >> For ZookeeperConsumerConnector in general, users need to wire in >>listener >> by themselves in code. >> For Mirror Maker, an internal rebalance listener has been wired in by >> default to avoid duplicates on consumer rebalance. User could still >> specify a custom listener class in command line argument, the internal >> rebalance listener will call that listener after it finishes the default >> logic. >>> >>>2. "The only source of truth for offsets in consume-then-send pattern >>>is end user." - I assume you don't mean an actual person, right? So >>>what does "end user" refer to? Can you clarify when will the offset >>>commit thread commit offsets? And which JIRA implements this? >> By end user I mean the target cluster here. The offset commit thread >> commit thread periodically. It only commit the offsets that have been >> acked. >>> >>>3. Maintaining message order - In which JIRA do we implement this part? >> KAFKA-1650 >>> >>>Again, thanks a lot for documenting this and even more for the >>>implementation - it is super important for many use cases. >>> >>>Gwen >>> >>> >>>Gwen >>> >>>On Tue, Jan 20, 2015 at 4:31 PM, Jiangjie Qin >>><j...@linkedin.com.invalid> >>>wrote: >>>> Hi Kafka Devs, >>>> >>>> We are working on Kafka Mirror Maker enhancement. A KIP is posted to >>>>document and discuss on the followings: >>>> 1. KAFKA-1650: No Data loss mirror maker change >>>> 2. KAFKA-1839: To allow partition aware mirror. >>>> 3. KAFKA-1840: To allow message filtering/format conversion >>>> Feedbacks are welcome. Please let us know if you have any questions or >>>>concerns. >>>> >>>> Thanks. >>>> >>>> Jiangjie (Becket) Qin >>
