users@kafka.apache.org

Hi Bharat, It looks authorization is not configured correctly. I suggest taking a look at the following blog post that configures authentication and authorization and includes a vagrant setup that you can test: http://www.confluent.io/blog/apache-kafka-security-authorization-authentication-encryp

RE: KafkaProducer 0.9.0.1 continually sends metadata requests

Hi Ismael, The request does succeed and the reason it keeps requesting is a check in the Sender.run(long now) method. public void run(long now) { Cluster cluster = metadata.fetch(); // get the list of partitions with data ready to send RecordAccumulator.ReadyCheckResu

Re: KafkaProducer 0.9.0.1 continually sends metadata requests

Thanks Phil. I added the information to the JIRA. This needs to be verified, but it may be possible that the fix for KAFKA-3306 will also fix this as we won't be requesting all the topics any longer. Ismael On Wed, Apr 13, 2016 at 10:35 AM, Phil Luckhurst wrote: > Hi Ismael, > > The request doe

Metrics for Log Compaction

Hi everyone, We are starting to use log compaction for one of our topics and I was wondering if there are any specific metrics exposed to monitor how often compaction took place and/or how many records (with some metadata related to partition) were deleted ? I looked at JMX metrics that Kafka

Re: [0.10.1.0-SNAPSHOT] KafkaProducer.close() not committing

So I rebuilt last night with the latest from master branch. Unfortunately same problem--producer doesn't seem to commit/close().  After inserting a few records (which *seem* to go ok), close() times out and no offsets are updated--still set to 0. Any ideas?Thanks,Greg From: Greg Zoller T

Console consumer group id question

So, I know I can put group.id in the consumer.config file, but I would like to reuse the same config file for multiple groups in testing. I *thought* this would work: kafka-console-consumer.sh --bootstrap-server --new-consumer --consumer.config --topic --property group.id= That doesn't pr

RE: What is the best way to publish and consume different type of messages?

I suggest creating your own wrapper object to hold the data. When using java serialized objects you'll need to take care of backwards compatibility. While this issue exists no matter what format you use, it is easy to make a new version of a java object and not think about if it is compatible.

Help understanding a failure please.

Hi all. I'm running a two node cluster that has been rock solid for almost a year and a half. We experienced an outage of one of the two brokers this morning, and from the logs, I'm not sure what happened, and how to prevent it. The Kafka version is 0.8.1.1 with Scala 2.10. Java version is Open

Kafka Connect - Source Connector for Apache Cassandra

Has anyone tried implementing or know whether it's possible to implement a "Kafka Connect" source connector for apache Cassandra, similar to its JDBCSourceConnector? Thanks.

Re: [0.10.1.0-SNAPSHOT] KafkaProducer.close() not committing

Hi Greg, It may be worth passing a callback to the `send` to see if there's any error (or calling `get` on the Future returned). Ismael On Wed, Apr 13, 2016 at 2:59 PM, Greg Zoller wrote: > So I rebuilt last night with the latest from master branch. > Unfortunately same problem--producer doesn

Re: Spikes in kafka bytes out (while bytes in remain the same)

Thanks for your response Asaf. I have 4 brokers. These measurements are from the kafka brokers. This measurement on this graph comes from Kafka. It is a sum across all 4 brokers of the metric: kafka.server.BrokerTopicMetrics.BytesInPerSec.1MinuteRate. But I also have a system metric which I fe

Re: Kafka Connect - Source Connector for Apache Cassandra

There is one being worked on here https://github.com/tuplejump/kafka-connect-cassandra/ On Wed, Apr 13, 2016 at 5:21 PM, Kaz Chehresa wrote: > Has anyone tried implementing or know whether it's possible to implement a > "Kafka Connect" source connector for apache Cassandra, similar to its > JDBC

Re: Help understanding a failure please.

Sorry, if this sounds lame, but can you ping or telnet? On Wed, Apr 13, 2016 at 9:55 AM, Chris Neal wrote: > Hi all. > > I'm running a two node cluster that has been rock solid for almost a year > and a half. We experienced an outage of one of the two brokers this > morning, and from the logs,

Re: Kafka Newbie question

Also, a newbie, using 0.9.0.1, I think you meant auto.offset.reset=earliest, did the OP have an intention to use his own commit strategy/management by setting enable.auto.commit=false? With the auto.offset.reset=earliest, a "new" consumer will get the earliest partition offsets and commit them and

Re: [0.10.1.0-SNAPSHOT] KafkaProducer.close() not committing

I did and now it hangs on the first send.  Here's my output: (hopefully won't destroy formatting).The "::: Writing 1 :::" should output a line like that for each of 6 writes, but only the first is called, then hangs until timeout. [pool-10-thread-1-ScalaTest-running-KafkaSpec] INFO org.apache.ka

Re: [0.10.1.0-SNAPSHOT] KafkaProducer.close() not committing

I hate email forums... mess up your code! Ok, so I finally managed to get it to run past the first one.  Not sure what I did... just kept trying.  Seems a bit flakey. When it ran all 6 it did 6 sends() then waited 5 seconds sleep I put in there before the close.  Then I saw 6 of these: java.lang.

Re: Console consumer group id question

The options can only be used to set "The properties to initialize the message formatter." You have several options, you could use different properties fils, with only the group.id being different. Another option is use a .properties.template, with a group.id=, and with a batch script first set the