Re: error recovery in multiple thread reading from Kafka with HighLevel api

Just did some testing.It seems that the rebalance will occur upon *zookeeper.session.timeout.ms . * *So yes, if one thread died, the left over messages will be picked up by other threads.* On Thu, Aug 7, 2014 at 5:36 PM, Chen Wang wrote: > Guozhang, > Just t

Re: Uniform Distribution of Messages for Topic Across Partitions Without Effecting Performance

In the new producer, a client can specify the partition number for each message. Then, any partitioning strategy can be implemented by the client. Thanks, Jun On Thu, Aug 7, 2014 at 1:37 PM, Bhavesh Mistry wrote: > The root of problem is consumer lag on one or two partition even with no op >

Re: error recovery in multiple thread reading from Kafka with HighLevel api

Guozhang, Just to make it clear: If I have 10 threads with the same consumer group id, read the topic T. The auto commit is turned off, and commitOffset is called only when the message is processed successfully. If thread 1 dies when processing message from partition P1, and the last offset is Offs

Re: error recovery in multiple thread reading from Kafka with HighLevel api

Yes, in that case you can turn of auto commit and call commitOffsets manually after processing is finished. commitOffsets() will only write the offset of the partitions that the consumer is currently fetching, so there is no need to coordinate this operation. On Thu, Aug 7, 2014 at 5:03 PM, Chen

Re: error recovery in multiple thread reading from Kafka with HighLevel api

But with the auto commit turned on, I am risking off losing the failed message, right? should I turn off the auto commit, and only commit the offset when the message is processed successfully..But that would require the coordination between threads in order to know what is the right timing to commi

Re: error recovery in multiple thread reading from Kafka with HighLevel api

Hello Chen, With high-level consumer, the partition re-assignment is automatic upon consumer failures. Guozhang On Thu, Aug 7, 2014 at 4:41 PM, Chen Wang wrote: > Folks, > I have a process started at specific time and read from a specific topic. > I am currently using the High Level API(cons

error recovery in multiple thread reading from Kafka with HighLevel api

Folks, I have a process started at specific time and read from a specific topic. I am currently using the High Level API(consumer group) to read from kafka(and will stop once there is nothing in the topic by specifying a timeout). i am most concerned about error recovery in multiple thread context

Re: consumer rebalance weirdness

Turn on GC logging (verbose time stamps) and see how long your pauses are. Sure, try increasing the timeout to see if it fixes the problem, but I would hesitate making that change permanent until you understand the problem better. You could also profile your consumer to see where it is spendin

Re: consumer rebalance weirdness

Yeah, it's possible that's happening (but no smoking gun). The main thing I'm seeing is that when it actually takes the time to process messages, it takes longer to get back to the ConsumerIterator for the next message. That alone seems to be the problem (does that make any sense)? I would have

Re: [New Feature Request] Ability to Inject Queue Implementation Async Mode

Policies for which messages to drop, retain, etc seem like something you should code in your application. I personally would not like to see this extra complexity added to Kafka. Philip -- http://www.philipotoole.com > On Aug 7, 2014, at 2:44 PM, Bhavesh Mistry

Re: [New Feature Request] Ability to Inject Queue Implementation Async Mode

Basically, requirement is to support message dropping policy in event when queue is full. When you get storm of data (in our case logging due to buggy application code), we would like to retain current message instead of first one in queue. We will mitigate this by rate limiting on producer side

RE: Apache webserver access logs + Kafka producer

PS you can also try just feeding the logs into a Kafka console producer by doing: TransferLog "| /opt/kafka/bin/kafka-console-producer.sh --topic apache --broker-list broker-1:9092" ErrorLog "| /opt/kafka/bin/kafka-console-producer.sh --topic apache-errors --broker-list broker-1:9092" You can

RE: Apache webserver access logs + Kafka producer

Check out my logstash-kafka project: https://github.com/joekiller/logstash-kafka I believe the plugin will be merged into logstash itself soon but for now you can make it yourself. I would suggest making your apache format in json in your apache config and then stream the data through the logs

Re: Uniform Distribution of Messages for Topic Across Partitions Without Effecting Performance

The root of problem is consumer lag on one or two partition even with no op ( read log and discard it) consumer . Our use case is very simple. Send all the log lines to Brokers. But under storm of data (due to exception or application error etc), one or two partition gets lags behind while other

Re: Apache webserver access logs + Kafka producer

Fluentd might work or simply configure rsyslog or syslog-ng on the box to watch the Apache log files, and send them to a suitable Producer (for example I wrote something that will accept messages from a syslog client, and stream them to Kafka.  https://github.com/otoolep/syslog-gollector) More

Re: consumer rebalance weirdness

A big GC pause in your application, for example, could do it. Philip   - http://www.philipotoole.com On Thursday, August 7, 2014 11:56 AM, Philip O'Toole wrote: I think the question is what in your consuming application could cause it not to check

Re: consumer rebalance weirdness

I think the question is what in your consuming application could cause it not to check in with ZK for longer than the timeout.   - http://www.philipotoole.com On Thursday, August 7, 2014 8:16 AM, Jason Rosenberg wrote: Well, it's possible that when

Re: delete topic ?

Hi Jason, You do want to wait for the next release as a lot of stability fixes are going into that. Tim On Thu, Aug 7, 2014 at 10:25 AM, Gwen Shapira wrote: > Looking at the delete topic patch, it looks like there were > significant modifications in the controller code to support that, so I > t

Re: delete topic ?

Looking at the delete topic patch, it looks like there were significant modifications in the controller code to support that, so I think you are out of luck. (https://reviews.apache.org/r/20745) On Thu, Aug 7, 2014 at 8:18 AM, Jason Rosenberg wrote: > Since the deletion stuff is now in trunk, wo

Re: delete topic ?

Since the deletion stuff is now in trunk, would be compatible to issue the command from a jar built from trunk, against a running 0.8.1.1 cluster? Or does the cluster also have to be running trunk? (I'm guessing it does :)). I have some topics I'd like to delete, but don't want to wait for 0.8.2

Re: consumer rebalance weirdness

Well, it's possible that when processing, it might take longer than the zookeeper timeout to process a message, intermittently. Would that cause a zookeeper timeout? (btw I'm usind 0.8.1.1). On Thu, Aug 7, 2014 at 2:30 AM, Clark Haskins wrote: > Is your application possibly timing out its zoo