Re: ClientUtils.fetchTopicMetadata reports smaller ISR than ZkUtils.getLeaderIsrAndEpochForPartition

Do you see any ISR churns on the brokers? You can check the ISR expand/shrink rate jmx. Thanks, Jun On Wed, Dec 4, 2013 at 3:53 PM, Ryan Berdeen wrote: > I'm working on some monitoring tools for Kafka, and I've seen a couple of > clusters get into a state where ClientUtils.fetchTopicMetadata

Re: kafka_2.8.0/0.8.0 pom seems invalid

Here's the jira: https://issues.apache.org/jira/browse/KAFKA-1163 "I was honestly not aware folks still used 2.8.0 and there have been talks about discontinuing that." Heh, that sounds amazing, considering that's the binary release version you've put up for download :) I am using that too for b

Re: ClientUtils.fetchTopicMetadata reports smaller ISR than ZkUtils.getLeaderIsrAndEpochForPartition

Hello Ryan, ClientUtils.fetchTopicMetadata reads the topic metadata from the brokers, which cache the metadata stored in from ZK (as the ground truth). And ZkUtils.getLeaderIsrAndEpochForPartition directly read from ZK. The cached data in brokers is not guaranteed to be strictly synchronized with

ClientUtils.fetchTopicMetadata reports smaller ISR than ZkUtils.getLeaderIsrAndEpochForPartition

I'm working on some monitoring tools for Kafka, and I've seen a couple of clusters get into a state where ClientUtils.fetchTopicMetadata will show that not all replicas are in the ISR. At the same time, ZkUtils.getLeaderIsrAndEpochForPartition will show that all all partitions are in the ISR, and

Re: kafka_2.8.0/0.8.0 pom seems invalid

Thanks Joe. If any 0.8.0 client is compatible with this release, I'll just upgrade my client. On Wed, Dec 4, 2013 at 1:59 PM, Joe Stein wrote: > There is a ticket for the 2.8.0 POM issue (I would send the link to it but > JIRA seems to be down so don't know the ticket off the top of my head). >

Re: kafka_2.8.0/0.8.0 pom seems invalid

There is a ticket for the 2.8.0 POM issue (I would send the link to it but JIRA seems to be down so don't know the ticket off the top of my head). The binary releases are only done using 2.8.0 but that really shouldn't matter since that is just the broker and your producers and consumers are commu

kafka_2.8.0/0.8.0 pom seems invalid

Hi All, I was trying to upgrade the kafka to 0.8 but I get an empty jar file for org.apache.kafka kafka_2.8.0 0.8.0 However org.apache.kafka kafka_2.8.2 0.8.0 is good for me. BTW from the download page I can only see kafka_2.8.0_0.8.0. Where can I download the scala

Re: Is there a way to get the offset of a consumer of a topic?

Simple tool I wrote to monitor 0.7 consumers. https://github.com/otoolep/stormkafkamon On Wed, Dec 4, 2013 at 12:49 PM, David DeMaagd wrote: > You can use either the MaxLag MBean (0.8): > > http://kafka.apache.org/documentation.html#monitoring > > Or the ConsumerOffsetChecker (0.7 or 0.8, can't

Re: Is there a way to get the offset of a consumer of a topic?

You can use either the MaxLag MBean (0.8): http://kafka.apache.org/documentation.html#monitoring Or the ConsumerOffsetChecker (0.7 or 0.8, can't seem to find a doc reference for it): ./kafka-run-class.sh kafka.tools.ConsumerOffsetChecker ... -- Dave DeMaagd | S'aite Reliability Engineering,

Re: Is there a way to get the offset of a consumer of a topic?

You can use ConsumerOffsetChecker https://github.com/apache/kafka/blob/0.8/core/src/main/scala/kafka/tools/ConsumerOffsetChecker.scala (thats a link to the 0.8 code, but its also in 0.7) it just prints current offsets to screen, but you can look at the code to see what you'd do to just the offsets

Is there a way to get the offset of a consumer of a topic?

Hi, I was wondering if there's an easy way to get the current offset of a specified consumer? What we need is something to check the consumer offset every 5 min. If it's changing at a certain rate every time we check, good. If not, then we have a problem. I was going to write a monitoring progr

RE: set new retention size on the fly

Great. Thanks, Jun. Regards, Libo -Original Message- From: Jun Rao [mailto:jun...@gmail.com] Sent: Wednesday, December 04, 2013 12:46 PM To: users@kafka.apache.org Subject: Re: set new retention size on the fly In 0.8.1, per topic config will only be persisted in ZK, not in the local

Re: Using Kafka 0.8 from Scala and Akka

Chetan, Are you also releasing a Scala RxJava producer as well? -Steve On Tue, Dec 3, 2013 at 10:42 PM, Richard Rodseth wrote: > Any update on this, Chetan? Thanks. > > > On Thu, Oct 31, 2013 at 4:11 PM, chetan conikee wrote: > > > I am in the process of releasing out Scala and RxJava consum

Re: set new retention size on the fly

In 0.8.1, per topic config will only be persisted in ZK, not in the local server property file. To change a topic config, you would do kafka-topics.sh --alter --config To change # partitons in a topic, you would do kafka-topics.sh --alter --partitions Our tentative future release schedule is

Re: Consumer not able to create message streams

Could you take a thread dump and see where the createMessageStreams is stuck? Thanks, Jun On Wed, Dec 4, 2013 at 4:10 AM, Tarang Dawer wrote: > Hi All > > I am using Kafka 0.8 , with inbuilt zookeeper. > I am runnings consumers in a jar. > > > > Configuration properties for creating consumerC

RE: set new retention size on the fly

Thanks for the clarification. I am just curious about how this works out. If we can change the retention size with "kafka-topics.sh --alter", will the new retention size be updated to the servers.properties? If there is any documentation, that will be great. Another question is whether you have an

Re: are topics and partitions dynamic?

"auto.create.topics.enable" are default true, so you don't need care about it. And you can set it false in the server.properties 2013/12/4 Magnus Edenhill > See the configuration reference here: > > https://kafka.apache.org/documentation.html#configuration > > > 2013/12/4 Arjun > > > hi, > >

Re: are topics and partitions dynamic?

See the configuration reference here: https://kafka.apache.org/documentation.html#configuration 2013/12/4 Arjun > hi, > > I was just looking at kafka 0.8, I could not find any option in > config/server.properties with key "auto.create.topics.enable" or > "default.replication.factor". > > Can

Re: are topics and partitions dynamic?

hi, I was just looking at kafka 0.8, I could not find any option in config/server.properties with key "auto.create.topics.enable" or "default.replication.factor". Can some one help me out,where i can find these. I want my topics to be created dynamically. thanks Arjun Narasimha Kota

Consumer not able to create message streams

Hi All I am using Kafka 0.8 , with inbuilt zookeeper. I am runnings consumers in a jar. Configuration properties for creating consumerConfig : - zookeeper.connect=IP group.id=consumerGroup fetch.message.max.bytes=10 zookeeper.session.timeout.ms=6 auto.offset.reset=smallest zookeepe