RE: steps path to kafka mastery

You can certainly try out things in a development environment. You could also think of contributing to the project itself as that will encourage you to develop deeper skills. But you are right. Kafka is pretty straightforward to setup and get running. The real skill IMO is in the ability to troubl

Re: Dealing with large messages

In addition to the config changes mentioned in that post, you may also have to change producer config if you are using the new producer. Specifically, *max.request.size* and *request.timeout.ms * have to be increased to allow the producer to send large messages. On 6 O

Re: Tools/recommendations to debug performance issues?

Have you checked the consumer lag? You can use the offset checker tool to see if there is a lag. On 14 Sep 2015 18:36, "noah" wrote: > We're using 0.8.2.1 processing maybe 1 million messages per hour. Each > message includes tracking information with a timestamp for when it was > produced, and a

Re: Question regarding to reconnect.backoff.ms

We did notice something similar. When a broker node (out of 3) went down, metadata calls continued to go to the failed node and producer kept failing. We were able to make it work by increasing the reconnect.backoff.ms to 1 second. Something similar was discussed earlier - http://qnalist.com/quest

Re: Kafka metadata

> > Alternatively you can get the same metadata from Zookeeper If this is true, why does the producer API makes it necessary to supply a value for metadata.broker.list? I noticed that this wasn't the case in 0.7. On 8 Aug 2015 04:06, "Lukas Steiblys" wrote: > Hi Qi, > > Yes, the metadata req

Re: How to monitor Kafka offset

Burrow works only if you are storing the offsets in kafka topic, not zookeeper. You can also take a look at Kafka web console ( it has a memory leak bug but a patch is available ). On 10 Jul 2015 09:34, "Jiangjie Qin" wrote: > You can take a look at Burrow. We use it in LinkedIn. > > Thanks, > >

Re: New producer: metadata update problem on 2 Node cluster.

tstrap" nodes in your current metadata set, even > if they aren't replicas for any of the topics you're working with. > > -Ewen > > > > On Thu, May 7, 2015 at 12:06 AM, Rahul Jain wrote: > > > Creating a new consumer instance *does not* solve this problem

Re: New producer: metadata update problem on 2 Node cluster.

, is this issue here specific to the "new" producer? With > > the "old" producer, we routinely construct a new producer which makes a > > fresh metadata request (via a VIP connected to all nodes in the cluster). > > Would this approach work with the new producer

Re: New producer: metadata update problem on 2 Node cluster.

might have came back up and allowed for a metadata update. Can you confirm > this? > > Thanks, > > Mayuresh > > On Tue, May 5, 2015 at 5:10 AM, Rahul Jain wrote: > > > We observed the exact same error. Not very clear about the root cause > > although it appears to

Re: New producer: metadata update problem on 2 Node cluster.

We observed the exact same error. Not very clear about the root cause although it appears to be related to leastLoadedNode implementation. Interestingly, the problem went away by increasing the value of reconnect.backoff.ms to 1000ms. On 29 Apr 2015 00:32, "Ewen Cheslack-Postava" wrote: > Ok, all