programmatically get number of items in topic/partition

Hi, How can I programmatically get the number of items in a topic, pending for consumption? If no programmatic way is avail, what other method is available? Shlomi

Re: Created topic by 2 partitions, only can use the one partition

Anyone has idea? Thank you Sent from my iPhone > On Sep 30, 2014, at 1:45 PM, Jiang Jacky wrote: > > Hi, Guys > It is very weird, I created a topic with 2 partitions couple weeks ago, and I > can only production the message to partition 0, not partition 1, but for now, > I created a new topi

Re: Created topic by 2 partitions, only can use the one partition

Do you have any errors in the logs? Are you using a partition key or is your KeyedMessage just topic and message? If your not using a partition key then then take a look at this https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Whyisdatanotevenlydistributedamongpartitionswhenapartitioningke

Re: Zookeeper reconnect failed due to 'state changed (Expired)'

I understand that, but even if the leader quickly (within a few seconds) rejoins the cluster? I had thought that ack=1 meant that messages not yet replicated from a broker that has a serious fatal failure (disk corruption, etc.) would be lost forever. But, as long as broker with the unreplicat

Re: Zookeeper reconnect failed due to 'state changed (Expired)'

But, as long as broker with the unreplicated messages comes back into the ISR, those messages should be replicated, no? Or, would the fact that the a new leader has been chosen for a partition, and new messages have been sent to that new leader, cause the offending broker to drop the unreplicated

Re: Zookeeper reconnect failed due to 'state changed (Expired)'

Ah! That makes so much sense, and is likely the cause of our lost messages. Thanks, I am now experimenting with other ack values. I’m a little worried about latency, especially since some of our producers send traffic across the Atlantic (we don’t use MirrorMaker :/ ). On Oct 1, 2014, at 10:

Re: LeaderNotAvailableException, although leader elected

Andras, Thanks for your feedback! In my opinion programmatic message sending must work out of the box on the first try, without any exceptions, warnings or the need for additional configuration. I'd be glad to support/contribute. I agree that the behavior of the producer for the first message o

Re: Connection reset by peer

Also, in a comment on this thread you mentioned that this is an expected exception This is expected during shutdown of a client since the server's attempts at sending any outstanding responses fails. This happens since the other endpoint of the socket connection is dead (the client). On Thu, Sep

RE: BadVersion state in Kafka Logs

We don’t have GC problem. zookeeper runs at 0% GC Kafka broker runs at 1-3% GC We don’t see many major GC's in our monitoring. We use Concurrent Mark and Sweep GC. -Original Message- From: Joe Stein [mailto:joe.st...@stealth.ly] Sent: Tuesday, September 30, 2014 6:41 PM To: users@kafka.

Re: programmatically get number of items in topic/partition

Take a look at ConsumerOffsetChecker. It does just that: print the offset and lag for each consumer and partition. You can either use that class directly, or use it as a guideline for your implementation On Wed, Oct 1, 2014 at 2:10 AM, Shlomi Hazan wrote: > Hi, > How can I programmatically get t

Re: programmatically get number of items in topic/partition

The other method is via command line bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --group *groupName* --zkconnect *zkServer:2181* Refer : https://cwiki.apache.org/confluence/display/KAFKA/System+Tools#SystemTools-ConsumerOffsetChecker https://apache.googlesource.com/kafka/+/0.8.0-bet

map reduce Outputformat KafkaOutputFormat

Hi All, After a map reduce computation I would like to send the results out to the a Kafka queue. Is there a bridge available that will let me publish the output to a Kafka queue. Similar to the TextOutputFormat... maybe a KafkaOutputFormat... I found one source https://github.com/kafka-dev/kafk

Reassigning Partition Failing

Hi All, I had a 0.8.1.1 Kafka Broker go down, and I was trying to use the reassign partition script to move topics off that broker. When I describe the topics, I see the following: Topic: mini__022active_120__33__mini Partition: 0 Leader: 2131118 Replicas: 2131118,2166601,2163421 Isr: 2131

kafka producer performance test

Hi, All I built a 3-node kafka cluster, I want to make performance test, I found someone post following thread, that is exactly the problem I have: - While testing kafka producer performance, I found 2 testing scripts. 1) performance testing script in kafka distribution bin/kafka-p

Re: kafka producer performance test

It is available with Kafka package containing the source code. Download the package, build it and run the above command. Regards, Ravi On Wed, Oct 1, 2014 at 7:55 PM, Sa Li wrote: > Hi, All > > I built a 3-node kafka cluster, I want to make performance test, I found > someone post following th

Re: kafka producer performance test

Hi, Ravi Thanks for reply, this is how I build the kafka package 0.8 $ git clone https://git-wip-us.apache.org/repos/asf/kafka.git $ cd /etc/kafka $ git checkout -b 0.8 remotes/origin/0.8 $ ./sbt update $ ./sbt package $ ./sbt assembly-package-dependency So I believe I already build it, but stil

Re: kafka producer performance test

Hi Sa, That script was developed with the new producer that is included on trunk. Checkout trunk and build and it should be there. -Jay On Wed, Oct 1, 2014 at 7:55 PM, Sa Li wrote: > Hi, All > > I built a 3-node kafka cluster, I want to make performance test, I found > someone post following t