Re: KafkaSpout

Hi harsha, Yes i did check with the above mentioned spoutconfig setup I think i need to bypass the in memory zookeeper which i am unable to do in local mode of storm. Any thoughts on that? Thanks, Siddharth ubale

Re: ZkClient bug can bring down broker/consumer on zookeeper push in EC2 environment

The above pull request didn't work perfectly. After a bunch of testing experiment, we decided that fixing zkclient itself isn't easy. So we decided to go with another workaround. We're expecting zookeeper 3.5.0 will be stabilized as soon as possible with the feature to refresh connections and the

Re: ZkClient bug can bring down broker/consumer on zookeeper push in EC2 environment

Thanks for reporting this issue. Agree that this is a problem for Kafka users using AWS. Please can you open a JIRA so we can keep track of this? On Sun, Aug 17, 2014 at 11:41 AM, Bae, Jae Hyeon wrote: > Recently, we found the serious ZkClient bug, actual Apache Zookeeper client > bug, which can

MBeans, dashes, underscores, and KAFKA-1481

Hi, *Problem:* Some Kafka 0.8.x MBeans have names composed of things like --. Note how dashes are used as delimiters. When and don't contain the delimiter character all is good if you want to extract parts of this MBean name by simply splitting on the delimiter character. The problem is that

Re: Need Document and Explanation Of New Metrics Name in New Java Producer on Kafka Trunk

Hi Bhavesh, Yes, if you look at https://issues.apache.org/jira/browse/KAFKA-1481 it describes this sort of a problem with various delimiters used by Kafka. As Jun suggested, I'll start a new thread and try to clearly explain the problem. Otis -- Monitoring * Alerting * Anomaly Detection * Central

Re: Need Document and Explanation Of New Metrics Name in New Java Producer on Kafka Trunk

Hi Jun, I also wanted to highlight the implication of having a dot "." as part of the topic and client.id with ".". Both topic name and client id need rule enforcement. JMX Bean names are not correct: eg: I have topics called "topic.dot" and client.id as "my.client". JMX Bean name is: ka

Re: Need Document and Explanation Of New Metrics Name in New Java Producer on Kafka Trunk

Yes, that description is not precise. We do allow dots in general. However, a topic can't be just "." or "..". Thanks, Jun On Mon, Sep 15, 2014 at 9:31 AM, Michael G. Noll < michael+st...@michael-noll.com> wrote: > Ah -- I only tested with dots "." > > However, haven't you said earlier in this

Re: no zk connection timeout in consumer config of kafka 0.7?

We have that in 0.7 too. It's called zk.connectiontimeout.ms. Thanks, Jun On Mon, Sep 15, 2014 at 12:55 AM, Yonghui Zhao wrote: > Hi, > > In http://kafka.apache.org/08/configuration.html, there are 2 parameter > about zk in consumer. > > zookeeper.session.timeout.ms 6000 Zookeeper session ti

Re: Kafka High Level Consumer

Thanks Joe Stein This worked :) On Fri, Sep 12, 2014 at 3:19 PM, Rahul Mittal wrote: > Hi , > Is there a way in kafka to read data from all topics, from a consumer > group without specifying topics in a dynamic way. > That is if new topics are created on kafka brokers the consumer group > should

Re: Migrating data from old brokers to new borkers question

distribution will be even based on the number of partitions. It is the same logic as AdminUtils. see https://github.com/airbnb/kafkat/blob/master/lib/kafkat/command/reassign.rb#L39 On Sun, Sep 14, 2014 at 6:05 PM, Neha Narkhede wrote: > This is great. Thanks for sharing! Does kafkat automaticall

Re: KafkaSpout

Siddharth, Did you try using SpoutConfig spoutConfig.forceFromStart = false spoutConfig.startOffsetTime = kafka.api.OffsetRequest.LatestTime() There are some good examples here on integrating storm and kafka https://github.com/miguno/kafka-storm-starter https://github.com/miguno/kafka-sto

Re: Need Document and Explanation Of New Metrics Name in New Java Producer on Kafka Trunk

Ah -- I only tested with dots "." However, haven't you said earlier in this thread that only "-" and "_" are allowed as special chars? This would exclude dots, although in the error message below they (dots) are said to be fine. Is this a bug or a feature? ;-) --Michael > On 14.09.2014, at

Re: Non-blocking High-Level Consumer

There isn't a very clean way to do this in the 0.8.x high level consumer. You can configure consumer.timeout.ms to a certain value so that the consumer's blocking iterator returns if no message arrives for consumer.timeout.ms. Thanks Neha On Mon, Sep 15, 2014 at 6:33 AM, Ivan Balashov wrote: >

Non-blocking High-Level Consumer

Hi, Is it possible to read all available messages with HLC in a non-blocking way? E.g. read all messages and not wait for more messages to appear in the topic. As far as I understand, currently one has to keep high-level consumer in a separate thread until it is shut down explicitly, but how can

KafkaSpout

Hi all, I am using a kafka spout for reading data from a producer into my storm topology. On the kafkaconfig : 1. When i set Kafka.forcefromstart = true .. it read from the beginning , however if i close my topology midway and run the topology again by setting Kafka.forcefromstart = false , it doe

Re: Setting log.default.flush.interval.ms and log.default.flush.scheduler.interval.ms

Thanks Neha and Jun for the pointers. We will try and evaluate this as well. Hemanth On Sat, Sep 13, 2014 at 4:45 AM, Neha Narkhede wrote: > Hemanth, > > Specifically, you'd want to monitor > kafka:type=kafka.SocketServerStats:getMaxProduceRequestMs and > kafka:type=kafka.LogFlushStats:getMaxFl

no zk connection timeout in consumer config of kafka 0.7?

Hi, In http://kafka.apache.org/08/configuration.html, there are 2 parameter about zk in consumer. zookeeper.session.timeout.ms 6000 Zookeeper session timeout. If the consumer fails to heartbeat to zookeeper for this period of time it is considered dead and a rebalance will occur. zookeeper.con