We're trying to set up mirror maker to mirror data from EU dc to US dc. The network delay is ~150 ms. In recent test; we realized that mirror maker is not keeping up with load and have a lag trending upward all the time.
What are configurations that can be tuned up to make it work for the higher throughput ? How to decide number of producer and consumer threads ? ( number of topic partitions / brokers ? ) *Environment* ( both source and destination cluster ) Kafka version 0.9 ( Cloudera 0.9.0.0+kafka2.0.0+188 ) queue.size = 10000 queue.byte.size = 100MB 2 brokers on source, 3 brokers on destination *Mirror maker configs :* Producer properties : request.timeout.ms=120000 block.on.buffer.full=TRUE retries=2000000000 acks=all Consumer properties: request.timeout.ms=120000 auto.offset.reset=latest enable.auto.commit=false We've configured 4 producer and consumer threads. There is no security set up as of now so it's all PLAINTEXT. We have 4 topics are white listed to sync from EU to US. Only one of them is high throughput. We also have a message handler to strip off some sensitive information from EU to US but it only works on a low thru put topic; the message handler still try to process the other topics but let it pass thru. Thanks, Sunil Parmar
