Re: Question on running Kafka Producer in Java environment

Pl check your classpath. Some jars might be missing. On Sat, Jan 17, 2015 at 7:41 AM, Su She wrote: > Hello Everyone, > > Thank you for the time and help. I had the Kafka Producer running, but am > having some trouble now. > > 1) Using Maven, I wrote a Kafka Producer similar to the one found her

Question on running Kafka Producer in Java environment

Hello Everyone, Thank you for the time and help. I had the Kafka Producer running, but am having some trouble now. 1) Using Maven, I wrote a Kafka Producer similar to the one found here: https://github.com/pppsunil/HelloKafka/blob/master/src/main/java/com/spnotes/kafka/HelloKafkaProducer.java 2)

Re: dumping JMX data

Hi Scott, http://sematext.com/spm is one of the services Joe was referring to. Jmxc is a command line tool that will dump JMX, see Sematext github account. Otis > On Jan 16, 2015, at 21:00, Scott Chapman wrote: > > Thanks, I actually ran into those already. I was hoping just to be able to

Re: [VOTE] 0.8.2.0 Candidate 1

Perhaps you can see if you can reproduce this issue and then file a jira. Thanks, Jun On Fri, Jan 16, 2015 at 4:39 PM, svante karlsson wrote: > Hmm, produce "msg/sec in rate" seems to be per broker and "produce msg/per > sec" should also be per broker and thus be related. The problem is that f

Re: Kafka Cluster Monitoring and Documentation of Internals (JMX Metrics) of Rejected Events

You can use the DumpLogSegments tool to see if a log segment is indeed corrupted. Thanks, Jun On Mon, Jan 12, 2015 at 2:04 PM, Bhavesh Mistry wrote: > Hi Kafka Team, > > I am trying to find out Kafka Internal and how a message can be corrupted > or lost at brokers side. > > I have refer to fol

Re: dumping JMX data

Thanks, I actually ran into those already. I was hoping just to be able to dump the JMX data plain and simple. I can consume it with other tools but I am mostly just trying to get the metrics in some format... any format. I have some limitations on what I can build/run, so hoping I can just levera

Re: Consumer questions

Hi, 1. In SimpleConsumer, you must keep track of the offsets in your application. In the example code, "readOffset" variable can be saved in redis/zookeeper. You should plugin this logic in your code. High Level Consumer stores the last read offset information in ZooKeeper. 2. You wil

Re: dumping JMX data

Here are some more tools for that https://cwiki.apache.org/confluence/display/KAFKA/JMX+Reporters depending on what you have in place and what you are trying todo different options exist. A lot of folks like JMX Trans. My favorite quick out of the box is using https://github.com/airbnb/kafka-stat

dumping JMX data

I appologize in advance for a noob question, just getting started with kafka, and trying to get JMX data from it. So, I had thought that running the JXMTool with no arguments would dump all the data, but it doesn't seem to return. I do know I can query for a specific Mbean name seems to work. But

Syslog Producer to Kafka

Hi, we just opened sourced a syslog producer to Kafka https://github.com/stealthly/go_kafka_client/tree/master/syslog. This server has a few features that folks might be interested in. Besides it producing your data to Kafka you can also configure it (via command line) to associate meta data with

Re: [VOTE] 0.8.2.0 Candidate 1

Hmm, produce "msg/sec in rate" seems to be per broker and "produce msg/per sec" should also be per broker and thus be related. The problem is that for a time period the graphs indicated that 1) messages where only produced to one broker 2) messages where produces to two brokers. when I restarted b

Re: KOYA vs. Samza?

Hey Otis, I think the key phrase in Samza's description is: "Apache Samza is a distributed stream processing framework" Kafka is not a stream processing framework. It's a message queue/broker system. Stream processing frameworks (e.g. Storm, Spark Streaming, Samza, etc) use message queueing/brok

Re: KOYA vs. Samza?

An analogy I use is that in comparison to classic MapReduce with its storage (HDFS) and compute (MR), Kafka is the storage and Samza is the compute. But HDFS does not run through YARN... So, in the same way that HDFS is run and launched separately from YARN (and hence Map-Reduce), Kafka is also r

Re: KOYA vs. Samza?

Hm. My understanding was that both are aimed at basically the same thing - Kafka on YARN. From Samza site: "Apache Samza is a distributed stream processing framework. It uses Apache Kafka for messaging, and Apache Hadoop YARN

Re: [VOTE] 0.8.2.0 Candidate 1

I did some quick tests and the mbean values look reasonable. On the producer side, produce msg/sec is actually for all brokers. Thanks, Jun On Fri, Jan 16, 2015 at 12:09 PM, svante karlsson wrote: > Disregard the previous message, it was send accidently.. > > Jun, > > I don't know if it was an

Re: [VOTE] 0.8.2.0 Candidate 1

Apache mailing list doesn't allow attachments. Could you file a jira and attach the graph and the description there? Thanks, Jun On Fri, Jan 16, 2015 at 12:09 PM, svante karlsson wrote: > Disregard the previous message, it was send accidently.. > > Jun, > > I don't know if it was an issue with

Consumer questions

Hi, I am following this link: https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+SimpleConsumer+Example for a consumer design using kafka_2.9.2 version 0.8.1.1 (what I can find in maven central). I have a couple of questions about the consumer. I checked the archives and didn't see these

Re: KOYA vs. Samza?

Hey Otis, I'm not terribly familiar with KOYA, but my understanding is that it's a tool for deploying Kafka brokers to YARN, and administering them. I don't think that it has any stream processing functionality built into it. As such, it seems to me that KOYA and Samza could be used together: you

KOYA vs. Samza?

Hi, I was wondering if anyone can compare and contrast KOYA and Samza? Thanks, Otis -- Monitoring * Alerting * Anomaly Detection * Centralized Log Management Solr & Elasticsearch Support * http://sematext.com/

Re: [VOTE] 0.8.2.0 Candidate 1

Disregard the previous message, it was send accidently.. Jun, I don't know if it was an issue with graphite or the mbean and have not seen it since - and we have tried of several cases of failover. That said, I have the feeling that it was a kafka issue and I'm a bit suspicious about the new mbe

Re: kafka brokers going down within 24 hrs

Tousif, I meant to say if kafka broker is going down often its better to analyze whats the root of cause of the crash. Using supervisord to monitor kafka broker is fine, sorry about the confusion. -Harsha On Fri, Jan 16, 2015, at 11:25 AM, Gwen Shapira wrote: > Those errors

Re: Kafka sending messages with zero copy

Hi Rajiv, Thanks for this proposal, it would be great if you can upload some implementation patch for the CAS idea and show some memory usage / perf differences. Guozhang On Sun, Dec 14, 2014 at 9:27 PM, Rajiv Kurian wrote: > Resuscitating this thread. I've done some more experiments and profi

Re: [VOTE] 0.8.2.0 Candidate 1

Jun, I don't know if it was an issue with graphite or the mbean but I have not seen it since - and we have tried of several cases of failover and this problem has only been seen once. That said, I have the feeling that it was a kafka issue and I'm a bit suspicious about the new mbeans. I attach

Re: kafka brokers going down within 24 hrs

Those errors are expected - if broker 10.0.0.11 went down, it will reset the connection and the other broker will close the socket. However, it looks like 10.0.0.11 crashes every two minutes? Do you have the logs from 10.0.0.11? On Thu, Jan 15, 2015 at 9:51 PM, Tousif wrote: > i'm using kafka 2.

Query regarding Kafka publishing

Hi , I have a general query - As per the code in Kafka producer the serialization happens before partitioning , Is my understanding correct ? If yes whats the reason for it ? Regards, Liju John

Re: [VOTE] 0.8.2.0 Candidate 1

Svante, I tested this out locally and the mbeans for those metrics do show up on startup. Can you reproduce the issue reliably? Also, is what you saw an issue with the mbean itself or graphite? Thanks, Jun On Fri, Jan 16, 2015 at 4:38 AM, svante karlsson wrote: > I upgrade two small test clus

Re: [VOTE] 0.8.2.0 Candidate 1

I upgrade two small test cluster and I had two small issues but I'm, not clear yet as to if those were an issue due to us using ansible to configure and deploy the cluster. The first issue could be us doing something bad when distributing the update (I updated, not reinstalled) but it should be ea