total.memory.bytes vs buffer.memory in 0.8.2.x

Hi there, I would like to get some help regarding the use of the two producer properties mentioned above. 1. The API documentation of Producer.send

Kafka Out Of Memory Error

Hello all, Recently I got incident with kafka cluster, I found OutOfMemoryError in kafka server log > WARN [ReplicaFetcherThread-0-2], Error in fetch Name: FetchRequest; > Version: 0; CorrelationId: 7041716; ClientId: ReplicaFetcherThread-0-2; > ReplicaId: 0; MaxWait: 500 ms; MinBytes: 1 bytes; Re

Best way to watch for leader changes?

Hi, In Kafka 0.8 Simple Consumer https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+SimpleConsumer+Example shows how to watch for leader changes and find a new one: if (fetchResponse.hasError()) { // go find new leader // ... Is this still the recommended way of finding new leaders in 0.9?

Re: Kafka 090 maven coordinate

> On 04 Nov 2015, at 02:33, Grant Henke wrote: > > Hi Fajar, > > Please see my response to a similar email here: > http://search-hadoop.com/m/uyzND1ifZt65CCBS > > If you still have questions, please do not hesitate to ask. > Are there any changes to client protocol? If yes, there can we find

Re: Authorization Engine For Kafka Related to KPI-11

Bhavesh I am from the Apache Ranger team, so let me answer the questions pertaining to Ranger… > 1) Is there any performance impact with Brokers/Producer/Consumer while using > Apache Ranger ? As such Ranger overhead is negligible. Specifically for Kafka, we did some more optimization for the

Re: One more Kafka Meetup hosted by LinkedIn in 2015 (this time in San Francisco) - does anyone want to talk?

Thanks Joe. +1 on curating the page. In the meantime, you will also find some old videos and slides in the comments of past meet ups: http://www.meetup.com/http-kafka-apache-org/. On Wed, Nov 4, 2015 at 4:40 AM, Joe Stein wrote: > They should all be on the user groups section of the confluence

Re: Consuming "backwards"?

This is an aancient thread, but I thought I'd point to http://blog.sematext.com/2015/11/04/kafka-real-time-stream-multi-topic-catch-up-trick/ which gives a short description of how we ended up implementing this. It seems to work well for us, but if there are better ways to do it, esp. now with

Controlled shutdown

Hi, does anyone know in what cases Kafka will take itself down? I have a cluster of 2 nodes that went down (not crashed) this night in a controlled and orderly shutdown as far as I can tell, except it wasn't controlled by me Thanks Vadim

Re: invoking kafka consumer as soon as message arrives in topic

Hi, though i don't fully understand your question, i'd like to comment on your code design. 1. it's better for you to run you consumer in a separate thread instead of main thread, because it blocks the execution main thread. 2. you created a thread for each message, this is very costly. if it take

Re: One more Kafka Meetup hosted by LinkedIn in 2015 (this time in San Francisco) - does anyone want to talk?

They should all be on the user groups section of the confluence page https://cwiki.apache.org/confluence/display/KAFKA/Kafka+papers+and+presentations for which there were video. It might need some curating but that is where it has been going so far. ~ Joe Stein On Tue, Nov 3, 2015 at 4:48 PM, Gra