Re: Programmatic Kafka version detection/extraction?

That would be great! On Mon, Nov 10, 2014 at 9:45 PM, Jun Rao wrote: > Otis, > > We don't have an api for that now. We can probably expose this as a JMX as > part of kafka-1481. > > Thanks, > > Jun > > On Mon, Nov 10, 2014 at 7:17 PM, Otis Gospodnetic < > otis.gospodne...@gmail.com> wrote: > > >

Re: Adding replicas to existing topic cause data loss in some partitions

sorry, I didn't pay attention to these log. I did it by running a script in the night, and received feedback in the morning, I found the server log overflow by printing OutOfRange exception, so didn't catch any clue. I'll take care next time. Anyway, thanks a lot. On Mon, Nov 10, 2014 at 9:59 AM,

Re: expanding cluster and reassigning parititions without restarting producer

Neha, I understand that the producer kafka.javaapi.producer.Producer shown in examples is old, and that a new producer (org.apache.kafka.clients.producer) is avail? is it available for 0.8.1.1? how does it work? does it have a trigger fired when partitions are added or does the producer refresh som

zookeeper snapshot files eat up disk space

Hi, My zookeeper 'dataLogDir' is eating up my disk with tons of snapshot files. what are these files? what files can I delete? are week old files disposable? This folder only gets bigger... How can I avoid blowing my disk? Thanks, Shlomi

Re: zookeeper snapshot files eat up disk space

http://zookeeper.apache.org/doc/r3.4.6/zookeeperAdmin.html#Ongoing+Data+Directory+Cleanup /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop

Re: zookeeper snapshot files eat up disk space

That looks like a complete answer. BUT just to be sure: it says "Automatic purging of the snapshots and corresponding transaction logs was introduced in version 3.4.0". using 0.8.1.1 means that I will have to purge manually, right? Is there some convention for kafka users? e.g.: delete all but last

Re: kafka test jars in sbt?

Thanks Joe, that does the trick with sbt and the 0.8.2.-beta-test jar. Regards, Markus Joe Crobak schrieb am 23:28 Sonntag, 9.November 2014: > > >For sbt, you need to use something like: > >"org.apache.kafka" %% "kafka" %"0.8.2-beta" % "test" classifier "test" > >That tells sbt to pull in

Re: Programmatic Kafka version detection/extraction?

Hi Jun, Sounds good. But is the version number stored anywhere from where it could be gotten? Thanks, Otis -- Monitoring * Alerting * Anomaly Detection * Centralized Log Management Solr & Elasticsearch Support * http://sematext.com/ On Tue, Nov 11, 2014 at 12:45 AM, Jun Rao wrote: > Otis, >

Re: zookeeper snapshot files eat up disk space

There is a sidecar jvm project called exhibitor that manages this for you.   It's from Netflix so it's a bit aws-centric but still a good source for how to manage those log files.  You may also want to look into some zk config settings to make sure your logs are not growing too large by truncatin

Re: JavaKafkaWordCount not working under Spark Streaming

Here's a simple working version. import com.google.common.collect.Lists; import org.apache.spark.SparkConf; import org.apache.spark.api.java.function.FlatMapFunction; import org.apache.spark.api.java.function.Function; import org.apache.spark.api.java.function.Function2; import org.apache.spark.a

Re: Programmatic Kafka version detection/extraction?

Currently, the version number is only stored in our build config file, gradle.properties. Not sure how we can automatically extract it and expose it in an mbean. How do other projects do this? Thanks, Jun On Tue, Nov 11, 2014 at 7:05 AM, Otis Gospodnetic < otis.gospodne...@gmail.com> wrote: > H

Re: change retention for a topic on the fly does not work

For those who might need to do the same thing, the command is bin/kafka-topics.sh --zookeeper localhost:2182 --alter --topic yourconfig --config retention.ms=17280 On Mon, Nov 10, 2014 at 4:46 PM, Chen Wang wrote: > Hey guys, > i am using kafka_2.9.2-0.8.1.1 > > bin/kafka-topics.sh --zooke

Re: Cannot connect to Kafka from outside of EC2

Hi Guozhang, I was wondering if you found anything wrong in the logs/ thanks On Fri, Nov 7, 2014 at 4:19 PM, Sameer Yami wrote: > Hi Guozhang, > > Attached are the two logs with debug enabled. > > Thanks! > > On Fri, Nov 7, 2014 at 2:09 PM, Sameer Yami wrote: > >> The version is kafka_2.10-0

Re: Cannot connect to Kafka from outside of EC2

Hi Sameer, I think apache mailing list has blocked your attachment. If it is too long to include in the email body could you paste it somewhere and give me the link? Guozhang On Tue, Nov 11, 2014 at 10:01 AM, Sameer Yami wrote: > Hi Guozhang, > > I was wondering if you found anything wrong in

Re: Programmatic Kafka version detection/extraction?

In Sqoop we do the following: Maven runs a shell script, passing the version as a parameter. The shell-script generates a small java class, which is then built with a Maven plugin. Our code references this generated class when we expose "getVersion()". Its complex and ugly, so I'm kind of hoping

Re: Programmatic Kafka version detection/extraction?

In Kite, we parse the version from the POM properites file that Maven builds into our jar for us: private String getVersion() { String location = "/META-INF/maven/org.kitesdk/kite-tools/pom.properties"; String version = "unknown"; InputStream pomPropertiesStream = null; try {

Re: Programmatic Kafka version detection/extraction?

If is maven artifact then you will get following pre-build property file from maven build called pom.properties under /META-INF/maven/groupid/artifactId/pom.properties folder. Here is sample: #Generated by Maven #Mon Oct 10 10:44:31 EDT 2011 version=10.0.1 groupId=com.google.guava artifactId=guava

Re: expanding cluster and reassigning parititions without restarting producer

The new producer is available in 0.8.2-beta (the most recent Kafka release). The old producer only detects new partitions at an interval configured by topic.metadata.refresh.interval.ms. This constraint is no longer true for the new producer and you would likely end up with an even distribution of

Re: Cannot connect to Kafka from outside of EC2

Is it ok, if I send you directly? On Tue, Nov 11, 2014 at 10:17 AM, Guozhang Wang wrote: > Hi Sameer, > > I think apache mailing list has blocked your attachment. If it is too long > to include in the email body could you paste it somewhere and give me the > link? > > Guozhang > > On Tue, Nov 11

Re: spikes in producer requests/sec

Hi Wes, are you monitoring librdkafka statistics as well? If so, are there any correlating spikes in the per-broker and per-partition statistics? Such as: - brokers..rtt.avg <--- broker round-trip-time (latency) - brokers..waitresp_cnt <-- requests in flight - topics..partitions..msgq_cnt <

Re: Programmatic Kafka version detection/extraction?

So it looks like we can use Gradle to add properties to manifest file and then use getResourceAsStream to read the file and parse it. The Gradle part would be something like: jar.manifest { attributes('Implementation-Title': project.name, 'Implementation-Version': project.v

Re: spikes in producer requests/sec

There are some fixes in 0.8.2-beta for periodic latency spikes if you are using acks=-1 in the producer. -Jay On Tue, Nov 11, 2014 at 10:50 AM, Wes Chow wrote: > > We're seeing periodic spikes in req/sec rates across our nodes. Our > cluster is 10 nodes, and the topic has a replication factor o

Re: expanding cluster and reassigning parititions without restarting producer

Just to extend what Neha said. The new producer also picks up the new partitions by refreshing the metadata periodically (controlled metadata.max.age.ms). The new producer distributes the data more evenly to all partitions than the old producer. Thanks, Jun On Tue, Nov 11, 2014 at 11:19 AM, Neha

Re: No longer supporting Java 6, if? when?

Perhaps relevant: Hadoop is moving toward dropping Java6 in next release. https://issues.apache.org/jira/browse/HADOOP-10530 On Thu, Nov 6, 2014 at 11:03 AM, Jay Kreps wrote: > Yeah it is a little bit silly that people are still using Java 6. > > I guess this is a tradeoff--being more conserva

Security in 0.8.2 beta

Hi, Is there a way to secure the topics created in Kafka 0.8.2 beta? The need is to ensure no one is asked to read data from the topic without authorization. Regards Kashyap

Re: Security in 0.8.2 beta

Nope. Here's the JIRA where we are still actively working on security, targeting 0.9: https://issues.apache.org/jira/browse/KAFKA-1682 Gwen On Tue, Nov 11, 2014 at 7:37 PM, Kashyap Mhaisekar wrote: > Hi, > Is there a way to secure the topics created in Kafka 0.8.2 beta? The need > is to ensure

Re: expanding cluster and reassigning parititions without restarting producer

Understood. Thank you guys. On Wed, Nov 12, 2014 at 4:48 AM, Jun Rao wrote: > Just to extend what Neha said. The new producer also picks up the new > partitions by refreshing the metadata periodically (controlled > metadata.max.age.ms). The new producer distributes the data more evenly to > all

Re: Security in 0.8.2 beta

Simply encrypt your messages with a PSK between producers and consumers. On Nov 12, 2014 4:38 AM, "Kashyap Mhaisekar" wrote: > Hi, > Is there a way to secure the topics created in Kafka 0.8.2 beta? The need > is to ensure no one is asked to read data from the topic without > authorization. > > Re

Re: Security in 0.8.2 beta

I know a few implements that do this "encrypt your messages with a PSK between producers and consumers". One of them actually writes the "encrypted " on a different topic foreach downstream consumer private key that can read the message. This way when you are consuming you consume from two topics 1