Re: Kafka Newbie question

2016-04-09 Thread Pradeep Bhattiprolu
Liquan , thanks for the response. By setting the auto commit to false do i have to manage queue offset manually ? I am running a multiple threads with each thread being a consumer, it would be complicated to manage offsets across threads, if i dont use kafka's automatic consumer group abstraction.

Re: Kafka safe Rolling restart

2016-04-09 Thread Yashodhan Kocharekar
Hi Rob, thanks for detailed reply , let me share more info on this, unfortunately we are in production running traffic of ~ 25,000 message/sec on Kafka *0.8.1.1* cluster , now *controlled.shutdown.enable = false* this property enables controlled shutdown behaviour but it is set to false by d

Re: .Net Kafka Clients

2016-04-09 Thread Fastmail
Hi Dave,           We have our users running this client https://github.com/Microsoft/Kafkanet.  Thanks,Harsha _ From: Tauzell, Dave Sent: Friday, April 8, 2016 11:39 AM Subject: .Net Kafka Clients To: We are about to embark on using Kafka and will be starting

Re: Kafka safe Rolling restart

2016-04-09 Thread Robert Christ
Hi Yashodhan, I do this quite frequently and if I understand your question correctly, it is the default behavior. If you issue a normal TERM signal to the kafka process (or call kafka-server-stop.sh) it will start controlled shutdown which will migrate leadership for all the partitions it is curr

Re: Kafka Newbie question

2016-04-09 Thread Liquan Pei
Hi Pradeep, Can you try to set enable.auto.commit = false if you want to read to the earliest offset? According to the documentation, auto.offset.reset controls what to do when there is no initial offset in Kafka or if the current offset does not exist any more on the server (e.g. because that dat