Becket Qin from LinkedIn spoke at a meetup about how to tune the Kafka producer. One scenario that he described was tuning for situations where you had high network latency. See slides at https://www.slideshare.net/mobile/JiangjieQin/producer-performance-tuning-for-apache-kafka-63147600 and video at https://youtu.be/oQe7PpDDdzA
-James Sent from my iPhone > On Jul 21, 2017, at 9:25 AM, Sunil Parmar <sunilosu...@gmail.com> wrote: > > We're trying to set up mirror maker to mirror data from EU dc to US dc. The > network delay is ~150 ms. In recent test; we realized that mirror maker is > not keeping up with load and have a lag trending upward all the time. > > What are configurations that can be tuned up to make it work for the higher > throughput ? > How to decide number of producer and consumer threads ? ( number of topic > partitions / brokers ? ) > > > *Environment* ( both source and destination cluster ) > > Kafka version 0.9 ( Cloudera 0.9.0.0+kafka2.0.0+188 ) > > queue.size = 10000 > queue.byte.size = 100MB > > 2 brokers on source, 3 brokers on destination > > > *Mirror maker configs :* > > Producer properties : > request.timeout.ms=120000 > block.on.buffer.full=TRUE > retries=2000000000 > acks=all > > > Consumer properties: > request.timeout.ms=120000 > auto.offset.reset=latest > enable.auto.commit=false > > We've configured 4 producer and consumer threads. > There is no security set up as of now so it's all PLAINTEXT. > > We have 4 topics are white listed to sync from EU to US. Only one of them > is high throughput. We also have a message handler to strip off some > sensitive information from EU to US but it only works on a low thru put > topic; the message handler still try to process the other topics but let it > pass thru. > > Thanks, > Sunil Parmar
