Re: How does one measure performance of an existing Kafka cluster?

It is but everyone has different systems to-do that with (Cacti, Graphite, Ganglia, Riemann, Etc) there are reporters available for them even as a service like from Sematext, Boundary and DataDog too. More of the ecosystem https://cwiki.apache.org/confluence/display/KAFKA/Ecosystem or maybe if you

RE: How does one measure performance of an existing Kafka cluster?

I'm surprised that doesn't already exist - I would think that would be a common requirement? -Original Message- From: Neha Narkhede [mailto:neha.narkh...@gmail.com] Sent: Thursday, February 27, 2014 11:41 PM To: users@kafka.apache.org Subject: Re: How does one measure performance of an ex

Re: How does one measure performance of an existing Kafka cluster?

The jmx beans will expose the metrics per broker. You would need some utility to aggregate across all brokers in a cluster. Thanks, Neha On Thu, Feb 27, 2014 at 8:31 PM, Dan Hoffman wrote: > Are the numbers for the entire cluster or just the broker connected to? > (I'm interested in the forme

Re: New Consumer API discussion

Thank you, Neha, that makes it clear. Really, the aspect of all this that we could really use is a way to do exactly once processing. We are looking at more critical data. What are the latest thoughts on how to achieve exactly once and how might that affect a consumer API? Thanks, Rob On Fe

Re: How does one measure performance of an existing Kafka cluster?

Are the numbers for the entire cluster or just the broker connected to? (I'm interested in the former) On Thu, Feb 27, 2014 at 11:26 PM, Jun Rao wrote: > You can take a look at the jmx in > http://kafka.apache.org/documentation.html#monitoring > > Thanks, > > Jun > > > On Thu, Feb 27, 2014 at

Re: How does one measure performance of an existing Kafka cluster?

You can take a look at the jmx in http://kafka.apache.org/documentation.html#monitoring Thanks, Jun On Thu, Feb 27, 2014 at 7:50 PM, Dan Hoffman wrote: > Let's say I have a running cluster and users/apps are pounding away at it. > Is there a quick and easy way to measure its current throughp

How does one measure performance of an existing Kafka cluster?

Let's say I have a running cluster and users/apps are pounding away at it. Is there a quick and easy way to measure its current throughput? I know there are utilities for generating volume to get stats, but I'd like to simply get some stats about its current operation. Is there a good way to do

Re: Does high level consumer support setting offset

Currently, the only way to rewind the high level consumer to a different offset is to first shutdown all instances of the consumer, update zookeeper with the desired offsets and then restart the consumer. We are addressing this in the new consumer APIs for Kafka 0.9. If you can review the proposal

Re: Does high level consumer support setting offset

Hello Yonghui, In 0.8 high level consumer does not support setting offsets directly. However, you can use this tool to manually set the offset in ZK before you re-start the consumer to change the starting offset: bin/kafka-run-class.sh kafka.tools.UpdateOffsetsInZK Guozhang On Thu, Feb 27, 201

Re: New Consumer API discussion

Is thiswhat you are looking for? Basically, I think from the overall feedback, it looks like code snippets don't seem to work for overall

Does high level consumer support setting offset

Hi， In kafka 0.8, does high level consumer support setting offset? Our service reads kafka data but won't flush the data immediately, so if restarted the data in memory will be lost. We want to reset kafka consumer offset to an old offset. If the consumer group has only 1 machine, we can recor

Re: can't connect to kafka from a java client.

zk.connect is needed in Kafka 0.7. Since you are using Kafka 0.8, you need to set metadata.broker.list. ZK is no longer needed in the producer in 0.8. You can follow http://kafka.apache.org/documentation.html for 0.8 documentation. The latest release in maven is 0.8.0. We publish different kafka j

can't connect to kafka from a java client.

Goal: I am trying to send messages to kafka from a java cleint. And it has been a pain.. Let me describe in brief. 1. I have installed kafka on a centos VM. 2. I ran the zookeeper that comes with it, the server , the producer and the client with all of the default properties files. I sent and

Re: New Consumer API discussion

Neha, I see how one might wish to implement onPartitionsAssigned and onPartitionsRevoked, but I don’t have a sense for how I might supply these implementations to a running consumer. What would the setup code look like to start a high-level consumer with these provided implementations? thanks

Re: Unable to consume Snappy compressed messages with Simple Consumer

that worked! My publisher is sending 1MB payload and compressing it with snappy. I would have thought that with compression that it would have fit into the 10 bytes default of the sample code. I guess not! Thanks. On Thu, Feb 27, 2014 at 1:37 AM, Jun Rao wrote: > Try making the last pa

Re: New Consumer API discussion

Rob, The use of the callbacks is explained in the javadoc here - http://people.apache.org/~nehanarkhede/kafka-0.9-consumer-javadoc/doc/kafka/clients/consumer/ConsumerRebalanceCallback.html Let me know if it makes sense. The hope is to improve the javadoc so that it is self explanatory. Thanks, N

Re: Kafka 0.8: delete topic with data

No, delete topic support doesn't exist in 0.8 and the inclusion of DeleteTopicCommand was overlooked when we cut the release. So using that command can cause unexpected issues in the cluster and we don't recommend you use it. Delete topic is available in beta in the upcoming 0.8.1 release but I thi

Re: Problems consuming snappy compressed messages via SimpleConsumer

I was actually referring to kafka-simple-consumer-shell.sh. Please make sure that you pass the same fetch size that you configured your high level consumer with, to the kafka-simple-consumer-shell with the --fetchsize option. Thanks, Neha On Wed, Feb 26, 2014 at 6:26 PM, Dan Hoffman wrote: > T

Re: Reg Partition and Replica?

Hi Bala, Partitions are what give Kafka parallelism and allow it to scale. Every message exists in exactly one partition. Replicas are exact copies of partitions on different machines. They allow Kafka to be reliable and not lose messages if a machine dies. So the answers are: 1. No, a messag

Re: Kafka-0.8 Log4j Appender

You can create a JIRA here - https://issues.apache.org/jira/browse/KAFKA. On Wed, Feb 26, 2014 at 9:33 PM, 김동경 wrote: > Actually, I am quite newbie to this. > What do you exactly want me to do? > You want me to raise an issue for this? > Then which JIRA can I access and what I should do? > >

Re: Reg Partition and Replica?

Maybe these pictures can help you: https://kafka.apache.org/images/log_anatomy.png http://www.michael-noll.com/blog/uploads/kafka-topics-partitions-replicas.png http://www.michael-noll.com/blog/uploads/kafka-cluster-overview.png And of course, this post might help too: http://www.michael-noll.co

Kafka 0.8: delete topic with data

Hi, I'm using Kafka 0.8 which does not have a command to delete topic. However, I need the functionality and I'm trying to adopt this approach: https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/admin/DeleteTopicCommand.scala. I see it simply deletes the topic node from ZK. My qu