Re: Random Broker/Partition for Producer

2013-07-31 Thread Jun Rao
I assume that this is for 0.7. It will do what you want, send your message to a random partition in a random broker. Thanks, Jun On Wed, Jul 31, 2013 at 1:47 PM, Jared Wong wrote: > Hi, > > I want to randomly send my messages to a random broker and a random > partition registered in Zookeeper

Re: Random Broker/Partition for Producer

2013-07-31 Thread Tejas Patil
Nope. The property "metadata.broker.list" in producer.properties [0] is just used to fetch the metadata about which broker owns which [topic-partition]. Based on this info, the producer data is routed to the relevant leader broker. By default Kafka uses DefaultPartitioner [1] which is used to rand

Re: reprocessing messages in kafka

2013-07-31 Thread Tejas Patil
As @Milind said, it is possible that a consumer consumes the same message more than once. This happens when there is an unclean shutdown of the consumer and it is not able to commit its latest offset to Zookeeper. When the failed consumer comes up, it would fetch the stale offset from zookeeper thu

Re: class kafka.common.LeaderNotAvailableException (kafka.producer.BrokerPartitionInfo)

2013-07-31 Thread Jun Rao
Any error in the broker's state-change log? Thanks, Jun On Wed, Jul 31, 2013 at 2:32 PM, Nandigam, Sujitha wrote: > I am running 0.8 beta version, I wrote a producer with > auto.create.topics.enable = true & type async , When I run the producer > class getting below > > [ProducerSendThread-]

Re: Unable to consume data because of below issue

2013-07-31 Thread Jun Rao
It seems that you turned on debug level logging and hit a format error during logging. Are you using the 0.8 beta1 release? I enabled logging there and didn't see that error. Thanks, Jun On Wed, Jul 31, 2013 at 10:56 AM, Seshadri, Balaji wrote: > Can you guys help me on this ?.Getting below e

Re: how to downgrade the metrics jar from 3.x to 2.2.0

2013-07-31 Thread Jun Rao
The version # indicates that it's an 0.8 jar. I don't know how old it is. About 3 months ago, the snapshot in 0.8 only works with metrics 3.x. We have since downgraded metrics to the latest stable release 2.2.0. Could you try the 0.8 beta1 release? That should work with metrics 2.2.0. Thanks, Ju

Random Broker/Partition for Producer

2013-07-31 Thread Jared Wong
Hi, I want to randomly send my messages to a random broker and a random partition registered in Zookeeper. If, in my producer configuration, I only specify zk.connect and I don't specify any broker.list or other information, will this happen? Best, Jared -- Jared Wong :: jar...@mit.edu

Moving raw logs from kafka into hadoop using camus

2013-07-31 Thread Vadim Keylis
Good afternoon. I am new to camus and would like to use it to move raw data from kafka to hadoop. Do I have to pre-create avro schema in advance or it automatically created for me? What is the role of the decoder class specified by camus.message.decoder.class in property file? Do I need to impl

Re: reprocessing messages in kafka

2013-07-31 Thread Milind Parikh
It is possible to consume the same message more than once with the same consumer. However WHAT you actually do with the message (such as idempotent writes) is the tricker part. Regards Milind On Wed, Jul 31, 2013 at 8:22 AM, Oleg Ruchovets wrote: > Hi , > > I just don't know which mail list is

RE:class kafka.common.LeaderNotAvailableException (kafka.producer.BrokerPartitionInfo)

2013-07-31 Thread Nandigam, Sujitha
I am running 0.8 beta version, I wrote a producer with auto.create.topics.enable = true & type async , When I run the producer class getting below [ProducerSendThread-] INFO Fetching metadata from broker id:0,host:localhost,port:9093 with correlation id 0 for 6 topic(s) INFO Fetching metada

Unable to consume data because of below issue

2013-07-31 Thread Seshadri, Balaji
Can you guys help me on this ?.Getting below error on High Level consumer.But topic does exist. {2013-07-31 11:52:57,176} WARN [UNITTEST_MERD7-181710-1375293172238-5f1b0917-leader-finder-thread] (Logging.scala:88) - [UNITTEST_MERD7-181710-1375293172238-5f1b0917-leader-finder-thread], Failed t

reprocessing messages in kafka

2013-07-31 Thread Oleg Ruchovets
Hi , I just don't know which mail list is correct to post this question( storm or kafka)? Sorry for cross post. I just read the documentation which describe guaranteed message processing with storm - https://github.com/nathanmarz/storm/wiki/Guaranteeing-message-processing. The question actua

RE: how to downgrade the metrics jar from 3.x to 2.2.0

2013-07-31 Thread Nandigam, Sujitha
I got the kafka_2.8.0-0.8-SNAPSHOT.jar from kafka-0.7/core/target/scala-2.8.0. -Original Message- From: Jun Rao [mailto:jun...@gmail.com] Sent: Wednesday, July 31, 2013 10:57 AM To: users@kafka.apache.org Subject: Re: how to downgrade the metrics jar from 3.x to 2.2.0 >From the code path

Re: how to downgrade the metrics jar from 3.x to 2.2.0

2013-07-31 Thread Jun Rao
>From the code path, this is clearly 0.8 code. How did you get the Kafka jar? Also, Kafka 0.8 only works with metrics 2.2.0. Thanks, Jun On Wed, Jul 31, 2013 at 6:52 AM, Nandigam, Sujitha wrote: > Yes its Kafka 0.7.0 and my jars in core/lib are > metrics-core-3.0.0-c0c8be71.jar and metrics-an

RE: how to downgrade the metrics jar from 3.x to 2.2.0

2013-07-31 Thread Nandigam, Sujitha
Yes its Kafka 0.7.0 and my jars in core/lib are metrics-core-3.0.0-c0c8be71.jar and metrics-annotation-3.0.0-c0c8be71.jar. I replaced them with metrics-core-2.2.0.jar metrics-annotation-2.2.0.jar. Please let me know how to replace without build errors.. Thanks, Sujitha -Original Message---