Re: KafkaProducer recovery/restart if broker dies

2015-08-31 Thread Alexey Sverdelov
Hi Erik, thank you for your help. I'll try your config today. Our topic has 80 partitions and replication factor of 2. Best regards, Alexey On Fri, Aug 28, 2015 at 5:39 PM, Helleren, Erik wrote: > Hi Alexey, > > So, a couple things. Your config seems to have some issues that would > result i

Re: KafkaProducer recovery/restart if broker dies

2015-08-28 Thread Kishore Senji
I faced the exact same problem recently. The JIRA is filed here: https://issues.apache.org/jira/browse/KAFKA-2459 Please have reconnect.backoff.ms to be greater than retry.backoff.ms (like 1sec more). I think the metadata expired and when it is trying to fetch the new metadata for this producer in

Re: KafkaProducer recovery/restart if broker dies

2015-08-28 Thread Helleren, Erik
Hi Alexey, So, a couple things. Your config seems to have some issues that would result in long wait times, You should try this configuration and see if you still have the issue: acks=1 compression.type=snappy retries=3 #Retry a few times to make it so they don¹t get dropped when a broker fails

KafkaProducer recovery/restart if broker dies

2015-08-28 Thread Alexey Sverdelov
Hi everyone, we run load tests against our web application (about 50K req/sec) and every time a kafka broker dies (also controlled shutdown), the producer tries to connect with the dead broker for about 10-15 minutes. For this time the application monitoring shows a constant error rate (about of 1