RE: NoBrokersForPartitionException

2013-07-15 Thread Sybrandy, Casey
Jun, Unfortunately, upgrades are slow to occur with respect to projects, so I don't know when this will occur. However, it looks like we will be upgrading over the next couple months to 0.7.2. Regardless, what is causing this? Is this a bug in Kafka or is it something that triggered by somet

Protocol:Offsetrespose

2013-07-15 Thread Vinicius Carvalho
Hi there, from the docs: "Time: Used to ask for all messages before a certain time (ms). There are two special values. Specify -1 to receive the latest offset (this will only ever return one offset). Specify -2 to receive the earliest available offsets." But when I send -1 as an offsetrequest I g

Re: OffsetResponse: protocol size error

2013-07-15 Thread Colin Blower
It looks like you may be parsing the result incorrectly. I would have expected your data structure to look like: If you are still having issues with this could you post the raw response bytes? { "topics": [ { "partitionOffsets": [ { "par

Re: OffsetResponse: protocol size error

2013-07-15 Thread Vinicius Carvalho
Yes, my bad. I need to pay more attention to the [] structures. Sorry On Mon, Jul 15, 2013 at 12:57 PM, Colin Blower wrote: > It looks like you may be parsing the result incorrectly. I would have > expected your data structure to look like: > > If you are still having issues with this could you

ConsumerRebalanceFailedException

2013-07-15 Thread Vaibhav Puranik
Hi all, We have a small Kafka cluster (0.7.1 - 3 nodes) in EC2. The load is about 200 million events per day, each being few kilobytes. We have a single node zookeeper. Yesterday suddenly our Kafka clients started throwing the following exception: java.lang.RuntimeException: kafka.common.Consumer

Re: 0.8.0-beta1 is now available in public maven

2013-07-15 Thread Chris Riccomini
Hey Joe, First off, thanks for doing this! I'm trying to use the publication with Gradle, and I'm running into problems. It's giving me a funky exception. I did some digging, and apparently the exception implies that Gradle has been given a 1.0 pom. I took a look at the Kafka pom: https://

Re: 0.8.0-beta1 is now available in public maven

2013-07-15 Thread Joe Stein
What is/are the error(s) you are getting? "intransitive" in the SBT line is so that SBT does not fetch the libraries what is set for exclusion in the POM an example of an error if you don't do use what should be excluded is in https://issues.apache.org/jira/browse/KAFKA-974 On Mon, Jul 15, 2013

Re: 0.8.0-beta1 is now available in public maven

2013-07-15 Thread Chris Riccomini
Hey Joe, The error I see is: 10:35:30.431 [ERROR] [org.gradle.BuildExceptionReporter] FAILURE: Build failed with an exception. 10:35:30.432 [ERROR] [org.gradle.BuildExceptionReporter] 10:35:30.432 [ERROR] [org.gradle.BuildExceptionReporter] * What went wrong: 10:35:30.432 [ERROR] [org.gradle.Buil

Re: 0.8.0-beta1 is now available in public maven

2013-07-15 Thread Joe Stein
I wonder if Gradle does not use Maven Central as the default repository like SBT does maybe you have to explicitly set this (shot in the dark, don't know Gradle). Maven Central Repository for resolver = http://repo1.maven.org/maven2/ On Mon, Jul 15, 2013 at 1:39 PM, Chris Riccomini wrote: > Hey

Re: 0.8.0-beta1 is now available in public maven

2013-07-15 Thread Chris Riccomini
Hey Joe, I can give that a shot. I did have to add the apache release repository to get kafka to resolve. I'll get back to you in a few. Cheers, Chris On 7/15/13 10:43 AM, "Joe Stein" wrote: >I wonder if Gradle does not use Maven Central as the default repository >like SBT does maybe you have

Re: 0.8.0-beta1 is now available in public maven

2013-07-15 Thread chetan conikee
I stumbled on a few issues when I was attempting to create a Scala based Client (Producer/Consumer) project. Finally worked after revisions: Here's the snippet of by build.sbt ~~ /* scala versions and options */ scalaVersion := "2.9.2" libraryDependencies += "org.apache.kafka" %

Re: 0.8.0-beta1 is now available in public maven

2013-07-15 Thread Chris Riccomini
Hey Guys, The problem persists, even when using the explicit URL Joe provided. I've also constructed a dummy Maven project, and tested Kafka dependencies there. That worked, but I had to explicit write the block. I might have to sick our local Gradle experts on this one... Cheers, Chris On 7/

Re: 0.8.0-beta1 is now available in public maven

2013-07-15 Thread Chris Riccomini
Hey Guys, Digging into this more. Here's a fun fact: Maven Central's POM does not match Apache release's. http://search.maven.org/remotecontent?filepath=org/apache/kafka/kafka_2.9.2 /0.8.0-beta1/kafka_2.9.2-0.8.0-beta1.pom https://repository.apache.org/content/groups/public/org/apache/

Re: 0.8.0-beta1 is now available in public maven

2013-07-15 Thread Joe Stein
hmmm, interesting I think the issue is that I had to-do multiple pushes to nexus to get this to work (unfortunately) and perhaps maven central did not overwrite like it did with apache rather appending and meshing the poms together. The POM in maven central looks like a combination of what I was

Re: Protocol:Offsetrespose

2013-07-15 Thread Colin Blower
So, thats my fault. The protocol documentation is wrong. I added that comment in the doc. The time -1 should specify latest offsets, which may be more than one. While the time -2 should specify the earliest offset and only ever return one. I have updated the documentation. Let me know if it is mo

Re: 0.8.0-beta1 is now available in public maven

2013-07-15 Thread Chris Riccomini
Hey Joe, Awesome. I've updated the Jira: https://issues.apache.org/jira/browse/KAFKA-974 Cheers, Chris On 7/15/13 11:24 AM, "Joe Stein" wrote: >hmmm, interesting > >I think the issue is that I had to-do multiple pushes to nexus to get this >to work (unfortunately) and perhaps maven centra

Consumer offset oscillates between 2 values

2013-07-15 Thread arathi maddula
Hi, I use a Java high level consumer to read from a topic with compression set to snappy. When I query the consumer offset for that group using zookeeper shell, the offset value oscillates between 2 values. I use kafka 0.8 and I never faced this issue before. This happens even when there are no hig

Re: 0.8.0-beta1 is now available in public maven

2013-07-15 Thread Jason Rosenberg
Any thoughts on my question, wrt scala version to prefer? Also, what of the double dependency on zookeeper? Should I file a jira for that? Jason On Sun, Jul 14, 2013 at 9:26 PM, Jason Rosenberg wrote: > Thanks for doing this! > > I'm wondering whether there is a reason to prefer one version

Leader not local for partition error?

2013-07-15 Thread Maxime Petazzoni
Hi all, I'm not sure if I'm doing something wrong or if I missed a step somewhere. A little while ago I successfully got the 0.8 quickstart example to work fine with the console producer/consumer. Then I went to work on some code to learn how to implement a producer, which failed with the producer

Re: Questions regarding broker

2013-07-15 Thread Joel Koshy
Hey Calvin, I apologize for not being able to get to this sooner. I don't think I can reproduce the full scenario exactly as I don't have exclusive access to so many machines, but I tried it locally and couldn't reproduce it. Any chance you can reproduce it with a smaller deployment? Is step 6 req

Re: ConsumerRebalanceFailedException when broker unavailable

2013-07-15 Thread Joel Koshy
I think this is addressed in KAFKA-969 On Fri, Jul 12, 2013 at 5:55 PM, Florin Trofin wrote: > Hello, > > I'm going through a series of tests with Kafka 0.8 to see what the failure > cases are. One simple scenario is this: > > 1. Launch Zookeeper > 2. Launch my consumer (written in scala) but do