Re: What is the benefit of using acks=all and minover e.g. acks=3

2015-11-27 Thread Gwen Shapira
Hi, min.insync.replica is alive and well in 0.9 :) Normally, you will have 4 our of 4 replicas in sync. However if one of the replicas will fall behind, you will have 3 out of 4 in sync. If you set min.insync.replica = 3, produce requests will fail if the number on in-sync replicas fall below 3.

Re: What is the benefit of using acks=all and minover e.g. acks=3

2015-11-27 Thread Prabhjot Bharaj
Hi Gwen, How about min.isr.replicas property? Is it still valid in the new version 0.9 ? We could get 3 out of 4 replicas in sync if we set it's value to 3. Correct? Thanks, Prabhjot On Nov 28, 2015 10:20 AM, "Gwen Shapira" wrote: > In your scenario, you are receiving acks from 3 replicas whil

Re: What is the benefit of using acks=all and minover e.g. acks=3

2015-11-27 Thread Gwen Shapira
In your scenario, you are receiving acks from 3 replicas while it is possible to have 4 in the ISR. This means that one replica can be up to 4000 messages (by default) behind others. If a leader crashes, there is 33% chance this replica will become the new leader, thereby losing up to 4000 messages

Re: Flush Messages in KafkaProducer Buffer

2015-11-27 Thread Guozhang Wang
0.9.0 producers are not compatible with 0.8.2 brokers, so you need to upgrade the brokers as well. You can find the upgrade guidance here: http://kafka.apache.org/documentation.html#upgrade Key point is to first upgrade the in-between broker protocol first before upgrading the broker and producer

Fair Usage of Kafka Queues

2015-11-27 Thread D
Hi, We have a ElasticSearch-Logstash-Kibana deployment in which multiple logstash-forwarders (from multiple service logs) pushes log to logstash which then sends it to Kafka and then one more logstash pulls those logs from Kafka and indexes them in ElasticSearch cluster. Right now we are usin

What is the benefit of using acks=all and minover e.g. acks=3

2015-11-27 Thread Andreas Flinck
Hi all The reason why I need to know is that we have seen an issue when using acks=all, forcing us to quickly find an alternative. I leave the issue out of this post, but will probably come back to that! My question is about acks=all and min.insync.replicas property. Since we have found a wor