Re: Error in acceptor (kafka.network.Acceptor)

Hi, it's probably beyond that. it may be an issue with the number of files Kafka can have opened concurrently. A previous conversation with Joe about (build failes for latest stable source tgz (kafka_2.9.2-0.8.1.1)) turned out to discuss this (Q's by Joe, A's by me): 1. what else on the logs? [*se

Php client for Kafka 0.8

Hi folks, One of things stopping us from upgrading version of Kafka from 0.7 to 0.8 is the absence of a native PHP client. A search points to this: https://github.com/salebab/phpkafka. Does anyone have experience with it ? We require a PHP client due to high throughput, low latency existing syste

Re: Manual Leader Assignment

> (2) > configure enough retries + backoff time in the producer (so that new > leaders can be elected during failure). Ya, I’ve been doing this, I need to get it just right. > (1) use ack=-1 in the producer; We use ack=1. If I configure (2) properly, I shouldn’t need ack=-1, will I? I don’t th

Kafka data can be stored on a Dbase instead of file system?

Hi Guys, I wanted to know if we can store data sent to consumers via brokers on to any database like how ActiveMQ persists data to kahaDB or any other Db like PostGRE,SQL etc. Also,when the broker is sending message to consumer and consumer fails, so when the consumer is up again will the consume

GenericJMX plugin file for Kafka 0.8.1

Hi, We plan to capture various metrics exposed in Kafka via JMX and wanted to check if anybody has a plugin file which could be reused. We found one such plugin file here but it seems to be targe

Re: GenericJMX plugin file for Kafka 0.8.1

Have you taken a look at https://cwiki.apache.org/confluence/display/KAFKA/JMX+Reporters or https://cwiki.apache.org/confluence/display/KAFKA/Ecosystem that might have more information that you are looking for. If you find something else that is not on the list if you could update it that would be

Re: GenericJMX plugin file for Kafka 0.8.1

We use jmxtrans to pull data out of JMX. https://github.com/wikimedia/puppet-kafka/blob/master/kafka-jmxtrans.json.md On Sep 9, 2014, at 7:54 AM, Navneet Gupta (Tech - BLR) wrote: > Hi, > > We plan to capture various metrics exposed in Kafka via JMX and wanted to > check if anybody has a p

Re: Kafka data can be stored on a Dbase instead of file system?

For your first question, I don't know of a way to do that. For the second question, consumers commit the offset after every/a few reads. When the consumer restarts, it starts from the last committed offset. On Tue, Sep 9, 2014 at 7:23 PM, siddharth ubale wrote: > Hi Guys, > > I wanted to know

Re: GenericJMX plugin file for Kafka 0.8.1

looks like I can't edit the wiki page. feel free to add https://github.com/airbnb/kafka-statsd-metrics2 to the page. thanks On Tue, Sep 9, 2014 at 8:10 AM, Andrew Otto wrote: > We use jmxtrans to pull data out of JMX. > > > https://github.com/wikimedia/puppet-kafka/blob/master/kafka-jmxtrans.jso

Re: zookeeper shows partition in-sync but local disk doesn't have the data dirs

noticed one important thing. topic foo's partition 1 and 2 have empty .log file on replicas. I suspect replication doesn't create the partition dir on broker 0 in this case, which then cause the WARN logs. On Mon, Sep 8, 2014 at 11:11 PM, Steven Wu wrote: > sorry. forgot to mention that I am ru

Introducing Bruce, a Kafka producer daemon

Hello Kafka users, Meet Bruce, a producer daemon developed at Tagged, Inc. (http://www.tagged.com). We are open sourcing Bruce because we have found it useful at Tagged, and believe others may also benefit from it. Bruce is available on GitHub (https://github.com/tagged/bruce ). We developed Bru

Re: Introducing Bruce, a Kafka producer daemon

This is very cool, Dave. Thanks to you and your team for this work. This may save us some work in the future. Special thanks for also paying such attention to monitoring! On Tue, Sep 9, 2014 at 12:24 PM, Dave Peterson wrote: > Hello Kafka users, > > Meet Bruce, a producer daemon developed at Ta

Re: Introducing Bruce, a Kafka producer daemon

Very cool, can you update the Wiki please? https://cwiki.apache.org/confluence/display/KAFKA/Clients maybe under a section called Daemon? Or something? /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly

Re: message size limit

HI Jun, Thanks for clarification. Follow up questions, does new producer solve the issues highlight. In event of compression and async mode in new producer, will it break down messages to this UPPER limit and submit or new producer strictly honor batch size. I am just asking if compression batc

Need Document and Explanation Of New Metrics Name in New Java Producer on Kafka Trunk

Kafka Team, Can you please let me know what each of following Metrics means ? Some of them are obvious, but some are hard to understand. My Topic name is *TOPIC_NAME*. can we enforce a Topic Name Convention or Metric Name Convention. Because in previous version of Kafka, we have similar issue

Re: zookeeper shows partition in-sync but local disk doesn't have the data dirs

Hmm, that seems to indicate that the leader info is not propagated properly from the controller to broker 0. In the state-change log of broker 0, do you see anything related to these 2 partitions when broker 0 comes up? Thanks, Jun On Tue, Sep 9, 2014 at 9:41 AM, Steven Wu wrote: > noticed one

Re: Need Document and Explanation Of New Metrics Name in New Java Producer on Kafka Trunk

Hi Bhavesh, Each of those JMX attributes comes with documentation. If you open up jconsole and attach to a jvm running the consumer you should be able to read the descriptions for each attribute. -Jay On Tue, Sep 9, 2014 at 2:07 PM, Bhavesh Mistry wrote: > Kafka Team, > > Can you please let me

Re: Error in acceptor (kafka.network.Acceptor)

Are you starting the broker in some container? You want to make sure that the container doesn't overwrite the open file handler limit. Thanks, Jun On Tue, Sep 9, 2014 at 12:05 AM, Shlomi Hazan wrote: > Hi, > it's probably beyond that. it may be an issue with the number of files > Kafka can hav

Re: Manual Leader Assignment

With ack=1, some acked msgs could be lost when the leader of a partition changes. Thanks, Jun On Tue, Sep 9, 2014 at 6:22 AM, Andrew Otto wrote: > > (2) > > configure enough retries + backoff time in the producer (so that new > > leaders can be elected during failure). > Ya, I’ve been doing th

Re: zookeeper shows partition in-sync but local disk doesn't have the data dirs

nope. sate-change log files only had some warnings regarding other partitions. nothing related to these two partitions. 2014-09-09 02:54:30,579 WARN kafka.utils.Logging$class:83 [kafka-request-handler-1] [warn] Broker 0 received invalid LeaderAndIsr request with correlation id 497 from controller

Re: message size limit

No, the new producer doesn't address that problem. Thanks, Jun On Tue, Sep 9, 2014 at 12:59 PM, Bhavesh Mistry wrote: > HI Jun, > > Thanks for clarification. Follow up questions, does new producer solve the > issues highlight. In event of compression and async mode in new producer, > will it

Re: zookeeper shows partition in-sync but local disk doesn't have the data dirs

In the controller log, do you see broker 0 being detected as the new broker when broker 0 comes up? Thanks, Jun On Tue, Sep 9, 2014 at 3:51 PM, Steven Wu wrote: > nope. sate-change log files only had some warnings regarding other > partitions. nothing related to these two partitions. > > 2014-

Re: zookeeper shows partition in-sync but local disk doesn't have the data dirs

assuming you are talking about controller log on broker 0, there is nothing there. $ less kafka-controller 2014-09-09 01:15:02,600 INFO kafka.utils.Logging$class:68 [main] [info] [ControllerEpochListener on 0]: Initialized controller epoch to 5 and zk version 4 2014-09-09 01:15:02,655 INFO kafka

Re: zookeeper shows partition in-sync but local disk doesn't have the data dirs

ah. maybe you mean the controller log on leader/controller broker 5. yes. I do noticed some errors regarding these two partitions. [2014-09-09 01:10:53,651] ERROR Controller 5 epoch 5 encountered error while changing partition [foo,2]'s state from New to Online since LeaderAndIsr path alrea dy ex

Re: message size limit

Hi Jun, Is there any plug-ability that Developer can customize batching logic or inject custom code for this ? Shall I file Jira for this issues. Thanks, Bhavesh On Tue, Sep 9, 2014 at 3:52 PM, Jun Rao wrote: > No, the new producer doesn't address that problem. > > Thanks, > > Jun > > On Tue,

Re: Need Document and Explanation Of New Metrics Name in New Java Producer on Kafka Trunk

Hi, > I have topic name with “.” So, it is hard to distinguish metric name and topic If your topics or hosts have dashes or underscores or your topic has dots you will have trouble extracting things from JMX. We provided a patch to change delimiter character in MBeans and are hoping it gets inc

Re: Need Document and Explanation Of New Metrics Name in New Java Producer on Kafka Trunk

Thanks, I was using without JMX. I will go through doc. But how about Topic or Metric name Topic Name Convention or Metric Name Convention ? The dot notation with topic having a ".". Any future plan to enforce some stand rules. Thanks, Bhavesh On Tue, Sep 9, 2014 at 3:38 PM, Jay Kreps wrote

Re: zookeeper shows partition in-sync but local disk doesn't have the data dirs

previous email is from state-change.log. also found this WARN in controller.log [2014-09-09 01:10:53,669] WARN [Controller 5]: Cannot remove replica 0 from ISR of partition [cdn_selection_runtime_data,8] since it is not in the ISR. Leader = 1 ; ISR = List(1) (kafka.controller.KafkaController) On

Re: Kafka data can be stored on a Dbase instead of file system?

Hi, Thanks Sharninder!! Does everyone else share the same view as sharninder that it is not possible to store data in database and we can only store data in configured folder at consumer end .please do let me know your views. Thanks, Siddharth On Tue, Sep 9, 2014 at 9:10 PM, Sharninder wrote:

Re: Kafka data can be stored on a Dbase instead of file system?

Based on the phrasing of your first question I might recommend taking either a closer look at how Kafka works. Kafka stores data on its broker servers in its own fashion as that is a key part of what makes it useful. It is not written to use another database for message storage. Consumers of the da