Re: Re: leader:none question

2013-10-30 Thread linghongbo008
 It seems only this way.  Thanks ! 
by the way, you have a plan to develop this functionality in a future version ?




From: Neha Narkhede
Date: 2013-10-30 13:13
To: users@kafka.apache.org; linghongbo008
Subject: Re: Re: leader:none question
We forgot to delete that script from the 0.8 beta distro. Since we don't have a 
way to delete the topic, the cleanest way to come out of this would be to wipe 
out the cluster and start fresh. If that's too much overhead, you can try 
bringing the cluster down, deleting the /brokers/topics/ data from 
zookeeper, delete the local directory from the brokers and restart the cluster. 
But we haven't tried the latter approach, so not sure if it works.

Thanks,
Neha




On Mon, Oct 28, 2013 at 7:03 PM, linghongbo008  wrote:

Hi, Neha !
I use kafka0.8  that offered to delete the function of the topic.
It can delete the topic by running the shell script "bin/kafka-delete-topic.sh "
and create the topic by running the script "kafka-create-topic.sh ".
Now, how do I fix this error to the topic of leader isn't  "leader :none"

Thanks.




From: Neha Narkhede
Date: 2013-10-28 22:36
To: users
Subject: Re: leader:none question

There is no way to delete topics in Kafka yet. So trying to delete topics
like this might cause failures and cause the topics to be in a bad state.

Thanks,
Neha
On Oct 28, 2013 4:46 AM, "linghongbo008"  wrote:

> Hi All
> I encountered an error.   I delete a certain topic and recreate it , but
> when I run the shell "kafka-list-topic.sh ",   the topic of all partitions
> "leader:none".
> I have no idea!  Please tell me what happened.
> Thanks.
>
>
> topic: mqttmsglog   partition: 0leader: nonereplicas: 11,13,14
>  isr:
> topic: mqttmsglog   partition: 1leader: nonereplicas: 12,14,11
>  isr:
> topic: mqttmsglog   partition: 2leader: nonereplicas: 13,11,12
>  isr:
> topic: mqttmsglog   partition: 3leader: nonereplicas: 14,12,13
>  isr:
> topic: mqttmsglog   partition: 4leader: nonereplicas: 11,14,12
>  isr:
> topic: mqttmsglog   partition: 5leader: nonereplicas: 12,11,13
>  isr:
> topic: mqttmsglog   partition: 6leader: nonereplicas: 13,12,14
>  isr:
> topic: mqttmsglog   partition: 7leader: nonereplicas: 14,13,11
>  isr:
> topic: mqttmsglog   partition: 8leader: nonereplicas: 11,12,13
>  isr:
> topic: mqttmsglog   partition: 9leader: nonereplicas: 12,13,14
>  isr:
> topic: mqttmsglog   partition: 10   leader: nonereplicas: 13,14,11
>  isr:

Topic related API

2013-10-30 Thread 小宇
Hi All,

why not provide  CreateTopic and DeleteTopic api, we know when
auto.create.topics.enable set true, kafka can auto create the topic, but
the problem is it always take some time before we can lookup the topic in
zookeeper, will this can be solved in the future version of kafka?

Thanks!
Best regards!


Re: Topic related API

2013-10-30 Thread 小宇
Anyone help?


2013/10/30 小宇 

> Hi All,
>
> why not provide  CreateTopic and DeleteTopic api, we know when
> auto.create.topics.enable set true, kafka can auto create the topic, but
> the problem is it always take some time before we can lookup the topic in
> zookeeper, will this can be solved in the future version of kafka?
>
> Thanks!
> Best regards!
>


Re: Topic related API

2013-10-30 Thread Viktor Kolodrevskiy
Why do you want to get topic list from zookeeper? As alternative you
can send TopicMetadataRequest to SimpleConsumer and get
TopicMetadataResponse that contains list of topics.

Thanks,
Viktor

2013/10/30 小宇 :
> Anyone help?
>
>
> 2013/10/30 小宇 
>
>> Hi All,
>>
>> why not provide  CreateTopic and DeleteTopic api, we know when
>> auto.create.topics.enable set true, kafka can auto create the topic, but
>> the problem is it always take some time before we can lookup the topic in
>> zookeeper, will this can be solved in the future version of kafka?
>>
>> Thanks!
>> Best regards!
>>



-- 
Thanks,
Viktor


Re: Re: leader:none question

2013-10-30 Thread Neha Narkhede
Yes, this will be included in either 0.8.1 or 0.8.2.

Thanks,
Neha
On Oct 30, 2013 1:09 AM, "linghongbo008"  wrote:

> **
>  It seems only this way.  Thanks !
> by the way, you have a plan to develop this functionality in a future
> version ?
> --
>
>  *From:* Neha Narkhede 
> *Date:* 2013-10-30 13:13
> *To:* users@kafka.apache.org; linghongbo008 
> *Subject:* Re: Re: leader:none question
>  We forgot to delete that script from the 0.8 beta distro. Since we don't
> have a way to delete the topic, the cleanest way to come out of this would
> be to wipe out the cluster and start fresh. If that's too much overhead,
> you can try bringing the cluster down, deleting the /brokers/topics/
> data from zookeeper, delete the local directory from the brokers and
> restart the cluster. But we haven't tried the latter approach, so not sure
> if it works.
>
> Thanks,
> Neha
>
>
> On Mon, Oct 28, 2013 at 7:03 PM, linghongbo008 wrote:
>
>> Hi, Neha !
>> I use kafka0.8  that offered to delete the function of the topic.
>> It can delete the topic by running the shell script
>> "bin/kafka-delete-topic.sh "
>> and create the topic by running the script "kafka-create-topic.sh ".
>> Now, how do I fix this error to the topic of leader isn't  "leader :none"
>>
>> Thanks.
>>
>>
>>
>>
>> From: Neha Narkhede
>> Date: 2013-10-28 22:36
>> To: users
>> Subject: Re: leader:none question
>>  There is no way to delete topics in Kafka yet. So trying to delete
>> topics
>> like this might cause failures and cause the topics to be in a bad state.
>>
>> Thanks,
>> Neha
>> On Oct 28, 2013 4:46 AM, "linghongbo008"  wrote:
>>
>> > Hi All
>> > I encountered an error.   I delete a certain topic and recreate it , but
>> > when I run the shell "kafka-list-topic.sh ",   the topic of all
>> partitions
>> > "leader:none".
>> > I have no idea!  Please tell me what happened.
>> > Thanks.
>> >
>> >
>> > topic: mqttmsglog   partition: 0leader: nonereplicas:
>> 11,13,14
>> >  isr:
>> > topic: mqttmsglog   partition: 1leader: nonereplicas:
>> 12,14,11
>> >  isr:
>> > topic: mqttmsglog   partition: 2leader: nonereplicas:
>> 13,11,12
>> >  isr:
>> > topic: mqttmsglog   partition: 3leader: nonereplicas:
>> 14,12,13
>> >  isr:
>> > topic: mqttmsglog   partition: 4leader: nonereplicas:
>> 11,14,12
>> >  isr:
>> > topic: mqttmsglog   partition: 5leader: nonereplicas:
>> 12,11,13
>> >  isr:
>> > topic: mqttmsglog   partition: 6leader: nonereplicas:
>> 13,12,14
>> >  isr:
>> > topic: mqttmsglog   partition: 7leader: nonereplicas:
>> 14,13,11
>> >  isr:
>> > topic: mqttmsglog   partition: 8leader: nonereplicas:
>> 11,12,13
>> >  isr:
>> > topic: mqttmsglog   partition: 9leader: nonereplicas:
>> 12,13,14
>> >  isr:
>> > topic: mqttmsglog   partition: 10   leader: nonereplicas:
>> 13,14,11
>> >  isr:
>>
>
>


Re: High level consumer stop reading data

2013-10-30 Thread Jun Rao
Are you running the latest code from the 0.8 branch?

Thanks,

Jun


On Tue, Oct 29, 2013 at 10:47 PM, Hanish Bansal <
hanish.bansal.agar...@gmail.com> wrote:

> No, it doesn't throw any exception. Consumer goes to halt state and after
> restarting it again starts consuming the data.
>
>
> On Wed, Oct 30, 2013 at 9:37 AM, Jun Rao  wrote:
>
> > Any exception/error from the consumer?
> >
> > Thanks,
> >
> > Jun
> >
> >
> > On Tue, Oct 29, 2013 at 4:50 AM, Hanish Bansal <
> > hanish.bansal.agar...@gmail.com> wrote:
> >
> > > Hi All,
> > >
> > > We are running kafka-0.8, If kafka node machine's network is restarted
> or
> > > lost for some time, then high level consumer stop reading data from
> kafka
> > > even after the network is restarted/working.
> > >
> > > --
> > > *Thanks & Regards*
> > > *Hanish Bansal*
> > >
> >
>
>
>
> --
> *Thanks & Regards*
> *Hanish Bansal*
>


Re: Plan for Scala 2.10+ support

2013-10-30 Thread Hisham Mardam-Bey
We have 0.7.x with Scala 2.10 and sbt 0.12 here if that's useful to anyone
else:

https://github.com/mate1/kafka/commits/0.7.2-m1-sbt-0.12


On Wed, Oct 30, 2013 at 12:08 AM, chetan conikee  wrote:

> Thanks Joe.
> We have been maintaining an 0.8+2.10 release in our private repo for the
> past few months.
>
>
> > On Oct 29, 2013, at 8:46 PM, Joe Stein  wrote:
> >
> > For now from the 0.8 branch you can build what your producer/consumer
> build
> > needs and publish to a local repository
> >
> > ./sbt "++2.10.2 publish-local"
> >
> > The 0.8.0 release vote is being held up with an error uploading/posting
> > artifacts https://issues.apache.org/jira/browse/INFRA-6927 and once
> that is
> > resolved we can continue towards having 0.8.0 in maven central with
> > supported builds
> >
> > /***
> > Joe Stein
> > Founder, Principal Consultant
> > Big Data Open Source Security LLC
> > http://www.stealth.ly
> > Twitter: @allthingshadoop 
> > /
> >
> >
> > On Tue, Oct 29, 2013 at 11:43 PM, Aniket Bhatnagar <
> > aniket.bhatna...@gmail.com> wrote:
> >
> >> I haven't been able to find one which we why had to build Kafka 0.8 from
> >> source. It's not super hard though to build Kafka.
> >>
> >>
> >>> On 30 October 2013 09:07, chetan conikee  wrote:
> >>>
> >>> Are there any public maven repos hosting 0.8 with 2.10+ support.?
> >>>
> >>> Sent from my iPhone
> >>>
> > On Oct 29, 2013, at 8:32 PM, Aniket Bhatnagar <
>  aniket.bhatna...@gmail.com> wrote:
> 
>  The latest 0.8 branch has support for scala 2.10. We use it in our
>  projects. Once 0.8 is released, I believe you should be able to see
> >> 2.10
>  artifacts in maven repository. Are you using 0.7 or 0.8?
> 
> 
> > On 30 October 2013 08:58, Abhinav Anand  wrote:
> >
> > Hi,
> > We are building pub-sub applications over kafka in our company. Some
> >> of
> > our packages have been build on Scala 2.10+. Though only Kafka_2.9.2
> >> is
> > available on maven repository. Kafka_2.9.2 uses some deprecated scala
> > classes (viz. ClassManifest), which causes runtime exception for our
> > application.
> >
> > We are currently maintaining our own repository of Kafka build over
> > scala_2.10. Though we wanted to open-source our system and wanted to
> >>> check
> > if there is any plan for Kafka_2.10 getting pushed to maven or apache
> > repository ??
> >
> > --
> > Abhinav Anand
> >>
>



-- 
Hisham Mardam-Bey
-=[ CTO ]-=-[ Mate1 Inc. ]=-

A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
A: Top-posting.
Q: What is the most annoying thing in e-mail?

-=[ Codito Ergo Sum ]=-


Questions about producer API

2013-10-30 Thread Roger Hoover
Hi,

I'm still getting started with Kafka and was curious why there is an
asymmetry between the producer and consumer APIs.  Why does the producer
config take a list of brokers where as the consumer config takes a list of
brokers?

Thanks,

Roger


Re: High level consumer stop reading data

2013-10-30 Thread Hanish Bansal
No. We are using binary of kafka "0.8.0 Beta1 Release".


On Wed, Oct 30, 2013 at 8:26 PM, Jun Rao  wrote:

> Are you running the latest code from the 0.8 branch?
>
> Thanks,
>
> Jun
>
>
> On Tue, Oct 29, 2013 at 10:47 PM, Hanish Bansal <
> hanish.bansal.agar...@gmail.com> wrote:
>
> > No, it doesn't throw any exception. Consumer goes to halt state and after
> > restarting it again starts consuming the data.
> >
> >
> > On Wed, Oct 30, 2013 at 9:37 AM, Jun Rao  wrote:
> >
> > > Any exception/error from the consumer?
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > >
> > > On Tue, Oct 29, 2013 at 4:50 AM, Hanish Bansal <
> > > hanish.bansal.agar...@gmail.com> wrote:
> > >
> > > > Hi All,
> > > >
> > > > We are running kafka-0.8, If kafka node machine's network is
> restarted
> > or
> > > > lost for some time, then high level consumer stop reading data from
> > kafka
> > > > even after the network is restarted/working.
> > > >
> > > > --
> > > > *Thanks & Regards*
> > > > *Hanish Bansal*
> > > >
> > >
> >
> >
> >
> > --
> > *Thanks & Regards*
> > *Hanish Bansal*
> >
>



-- 
*Thanks & Regards*
*Hanish Bansal*


Re: Plan for Scala 2.10+ support

2013-10-30 Thread Ngu, Bob
I am using Maven to build my Scala app but to resolve the bad POM
definition issue, https://issues.apache.org/jira/browse/KAFKA-1064, I
added all the required dependencies to my POM and thought I had to use
Scala 2.9.2 but it seems to build just fine with 2.10.2, this should be
fine, right?

Thanks,
Bob

On 10/30/13, 9:16 AM, "Hisham Mardam-Bey"  wrote:

>We have 0.7.x with Scala 2.10 and sbt 0.12 here if that's useful to anyone
>else:
>
>https://github.com/mate1/kafka/commits/0.7.2-m1-sbt-0.12
>
>
>On Wed, Oct 30, 2013 at 12:08 AM, chetan conikee 
>wrote:
>
>> Thanks Joe.
>> We have been maintaining an 0.8+2.10 release in our private repo for the
>> past few months.
>>
>>
>> > On Oct 29, 2013, at 8:46 PM, Joe Stein  wrote:
>> >
>> > For now from the 0.8 branch you can build what your producer/consumer
>> build
>> > needs and publish to a local repository
>> >
>> > ./sbt "++2.10.2 publish-local"
>> >
>> > The 0.8.0 release vote is being held up with an error
>>uploading/posting
>> > artifacts https://issues.apache.org/jira/browse/INFRA-6927 and once
>> that is
>> > resolved we can continue towards having 0.8.0 in maven central with
>> > supported builds
>> >
>> > /***
>> > Joe Stein
>> > Founder, Principal Consultant
>> > Big Data Open Source Security LLC
>> > http://www.stealth.ly
>> > Twitter: @allthingshadoop 
>> > /
>> >
>> >
>> > On Tue, Oct 29, 2013 at 11:43 PM, Aniket Bhatnagar <
>> > aniket.bhatna...@gmail.com> wrote:
>> >
>> >> I haven't been able to find one which we why had to build Kafka 0.8
>>from
>> >> source. It's not super hard though to build Kafka.
>> >>
>> >>
>> >>> On 30 October 2013 09:07, chetan conikee  wrote:
>> >>>
>> >>> Are there any public maven repos hosting 0.8 with 2.10+ support.?
>> >>>
>> >>> Sent from my iPhone
>> >>>
>> > On Oct 29, 2013, at 8:32 PM, Aniket Bhatnagar <
>>  aniket.bhatna...@gmail.com> wrote:
>> 
>>  The latest 0.8 branch has support for scala 2.10. We use it in our
>>  projects. Once 0.8 is released, I believe you should be able to see
>> >> 2.10
>>  artifacts in maven repository. Are you using 0.7 or 0.8?
>> 
>> 
>> > On 30 October 2013 08:58, Abhinav Anand 
>>wrote:
>> >
>> > Hi,
>> > We are building pub-sub applications over kafka in our company.
>>Some
>> >> of
>> > our packages have been build on Scala 2.10+. Though only
>>Kafka_2.9.2
>> >> is
>> > available on maven repository. Kafka_2.9.2 uses some deprecated
>>scala
>> > classes (viz. ClassManifest), which causes runtime exception for
>>our
>> > application.
>> >
>> > We are currently maintaining our own repository of Kafka build
>>over
>> > scala_2.10. Though we wanted to open-source our system and wanted
>>to
>> >>> check
>> > if there is any plan for Kafka_2.10 getting pushed to maven or
>>apache
>> > repository ??
>> >
>> > --
>> > Abhinav Anand
>> >>
>>
>
>
>
>-- 
>Hisham Mardam-Bey
>-=[ CTO ]-=-[ Mate1 Inc. ]=-
>
>A: Because it messes up the order in which people normally read text.
>Q: Why is top-posting such a bad thing?
>A: Top-posting.
>Q: What is the most annoying thing in e-mail?
>
>-=[ Codito Ergo Sum ]=-



Re: High level consumer stop reading data

2013-10-30 Thread Neha Narkhede
Could you try using 0.8 HEAD? We fixed quite a few consumer bugs from 0.8.0
Beta1 to 0.8 HEAD.

Thanks,
Neha


On Wed, Oct 30, 2013 at 10:29 AM, Hanish Bansal <
hanish.bansal.agar...@gmail.com> wrote:

> No. We are using binary of kafka "0.8.0 Beta1 Release".
>
>
> On Wed, Oct 30, 2013 at 8:26 PM, Jun Rao  wrote:
>
> > Are you running the latest code from the 0.8 branch?
> >
> > Thanks,
> >
> > Jun
> >
> >
> > On Tue, Oct 29, 2013 at 10:47 PM, Hanish Bansal <
> > hanish.bansal.agar...@gmail.com> wrote:
> >
> > > No, it doesn't throw any exception. Consumer goes to halt state and
> after
> > > restarting it again starts consuming the data.
> > >
> > >
> > > On Wed, Oct 30, 2013 at 9:37 AM, Jun Rao  wrote:
> > >
> > > > Any exception/error from the consumer?
> > > >
> > > > Thanks,
> > > >
> > > > Jun
> > > >
> > > >
> > > > On Tue, Oct 29, 2013 at 4:50 AM, Hanish Bansal <
> > > > hanish.bansal.agar...@gmail.com> wrote:
> > > >
> > > > > Hi All,
> > > > >
> > > > > We are running kafka-0.8, If kafka node machine's network is
> > restarted
> > > or
> > > > > lost for some time, then high level consumer stop reading data from
> > > kafka
> > > > > even after the network is restarted/working.
> > > > >
> > > > > --
> > > > > *Thanks & Regards*
> > > > > *Hanish Bansal*
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > *Thanks & Regards*
> > > *Hanish Bansal*
> > >
> >
>
>
>
> --
> *Thanks & Regards*
> *Hanish Bansal*
>


Kafka 0.8, Scala 2.10.2, OSGi

2013-10-30 Thread Ngu, Bob
Hi,

I decided to start a new thread regarding Kafka 0.8 beta1 support for Scala 
2.10.2 and OSGi. I asked about 2.10.2 in a different thread

I am using Maven to build my Scala library but to resolve the bad POM
definition issue, https://issues.apache.org/jira/browse/KAFKA-1064, I
added all the required dependencies to my POM and thought I had to use
Scala 2.9.2 but it seems to build just fine with 2.10.2, this should be
fine, right?

However, I am getting java.lang.NoClassDefFoundError: 
scala/reflect/ClassManifest error calling ProducerConfig
  val config = new ProducerConfig(props)

I believe this is because Kafka is built with 2.9.2 so I made the following 
changes to Build.scala
  scalaVersion := “2.10.2” ,
  publishTo := Some(Resolver.file("file",  new 
File(Path.userHome.absolutePath+"/.m2/repository"))),

I should be able to run “sbt publish” to my local maven repository. However, I 
also need to package this library in OSGi format since I am using the library 
in OSGi, does anyone know if this works in OSGi? I would also need to include 
addSbtPlugin("com.typesafe.sbt" % "sbt-osgi" % "0.4.0”) for OSGi bundling.

Thanks for any help!
Bob





Re: Kafka 0.8, Scala 2.10.2, OSGi

2013-10-30 Thread Ngu, Bob
Also I have to upgrade scalatest to 1.9 because 1.8 no longer exists.
However, I got a bunch of compile errors building with 2.10.2, any
insights will be much appreciated, thanks.

[error] 
/Users/rngu/ij/kafka-0.8.0-beta1-src/core/src/main/scala/kafka/utils/Annota
tions.scala:31: not found: type StaticAnnotation
[error] class nonthreadsafe extends StaticAnnotation
[error] ^
[error] 
/Users/rngu/ij/kafka-0.8.0-beta1-src/core/src/main/scala/kafka/api/FetchReq
uest.scala:166: type mismatch;
[error]  found   : kafka.utils.nonthreadsafe
[error]  required: scala.annotation.Annotation
[error] @nonthreadsafe
[error]  ^
[error] 
/Users/rngu/ij/kafka-0.8.0-beta1-src/core/src/main/scala/kafka/utils/Pool.s
cala:74: value asSet is not a member of object
scala.collection.JavaConversions
[error]   def keys = JavaConversions.asSet(pool.keySet())
[error]  ^
[error] 
/Users/rngu/ij/kafka-0.8.0-beta1-src/core/src/main/scala/kafka/utils/Annota
tions.scala:26: not found: type StaticAnnotation
[error] class threadsafe extends StaticAnnotation
[error]  ^
[error] 
/Users/rngu/ij/kafka-0.8.0-beta1-src/core/src/main/scala/kafka/consumer/Con
sumerTopicStats.scala:25: type mismatch;
[error]  found   : kafka.utils.threadsafe
[error]  required: scala.annotation.Annotation
[error] @threadsafe
[error]  ^
[error] 
/Users/rngu/ij/kafka-0.8.0-beta1-src/core/src/main/scala/kafka/consumer/Sim
pleConsumer.scala:28: type mismatch;
[error]  found   : kafka.utils.threadsafe
[error]  required: scala.annotation.Annotation
[error] @threadsafe
[error]  ^
[error] 
/Users/rngu/ij/kafka-0.8.0-beta1-src/core/src/main/scala/kafka/consumer/Zoo
keeperConsumerConnector.scala:443: value findIndexOf is not a member of
List[String]
[error]   val myConsumerPosition = curConsumers.findIndexOf(_ ==
consumerThreadId)
[error] ^
[error] val currentChildren =
JavaConversions.asBuffer(children).toSet
[error]   ^
[error] 
/Users/rngu/ij/kafka-0.8.0-beta1-src/core/src/main/scala/kafka/javaapi/Fetc
hRequest.scala:32: value asMap is not a member of object
scala.collection.JavaConversions
[error] val scalaMap = JavaConversions.asMap(requestInfo).toMap
[error]^
[error] 
/Users/rngu/ij/kafka-0.8.0-beta1-src/core/src/main/scala/kafka/javaapi/Offs
etRequest.scala:31: value asMap is not a member of object
scala.collection.JavaConversions
[error] val scalaMap = JavaConversions.asMap(requestInfo).toMap
[error]^
[error] 
/Users/rngu/ij/kafka-0.8.0-beta1-src/core/src/main/scala/kafka/javaapi/Topi
cMetadata.scala:20: value asList is not a member of object
scala.collection.JavaConversions
[error] import scala.collection.JavaConversions.asList
[error]^
[error] 
/Users/rngu/ij/kafka-0.8.0-beta1-src/core/src/main/scala/kafka/javaapi/Topi
cMetadata.scala:25: not found: value asList
[error] asList(topicMetadataSeq.map(new
kafka.javaapi.TopicMetadata(_)))
[error] ^
[error] 
/Users/rngu/ij/kafka-0.8.0-beta1-src/core/src/main/scala/kafka/javaapi/Topi
cMetadata.scala:29: not found: value asList
[error] asList(partitionMetadataSeq.map(new
kafka.javaapi.PartitionMetadata(_)))
[error] ^
[error] 
/Users/rngu/ij/kafka-0.8.0-beta1-src/core/src/main/scala/kafka/javaapi/Topi
cMetadata.scala:54: not found: value asList
[error]   def replicas: java.util.List[Broker] =
asList(underlying.replicas)
[error]  ^
[error] 
/Users/rngu/ij/kafka-0.8.0-beta1-src/core/src/main/scala/kafka/javaapi/Topi
cMetadata.scala:56: not found: value asList
[error]   def isr: java.util.List[Broker] = asList(underlying.isr)
[error] ^
[error] 
/Users/rngu/ij/kafka-0.8.0-beta1-src/core/src/main/scala/kafka/javaapi/Topi
cMetadataRequest.scala:30: value asBuffer is not a member of object
scala.collection.JavaConversions
[error] new kafka.api.TopicMetadataRequest(versionId, correlationId,
clientId, JavaConversions.asBuffer(topics))
[error]
^
[error] 
/Users/rngu/ij/kafka-0.8.0-beta1-src/core/src/main/scala/kafka/javaapi/cons
umer/SimpleConsumer.scala:28: type mismatch;
[error]  found   : kafka.utils.threadsafe
[error]  required: scala.annotation.Annotation
[error] @threadsafe
[error]  ^
[error] 
/Users/rngu/ij/kafka-0.8.0-beta1-src/core/src/main/scala/kafka/javaapi/cons
umer/ZookeeperConsumerConnector.scala:21: value asList is not a member of
object scala.collection.JavaConversions
[error] import scala.collection.JavaConversions.asList
[error]^
[error] 
/Users/rngu/ij/kafka-0.8.0-beta1-src/core/src/main/scala/kafka/javaapi/cons
umer/ZookeeperConsumerConnector.scala:76: not found: value asMap
[error] val scalaTopicCountMap: Map[String, Int] = Map.empty[String,
Int] ++ asMap(topicCountMap.asInstanceOf[java.util.Map

Re: Kafka 0.8, Scala 2.10.2, OSGi

2013-10-30 Thread Joe Stein
Can you go off the 0.8 branch please? We're in the process of the 0.8.0 release 
so if you still have problems that were not resolved we can know what they are.


/***
 Joe Stein
 Founder, Principal Consultant
 Big Data Open Source Security LLC
 http://www.stealth.ly
 Twitter: @allthingshadoop
/


On Oct 30, 2013, at 6:26 PM, "Ngu, Bob"  wrote:

> Hi,
> 
> I decided to start a new thread regarding Kafka 0.8 beta1 support for Scala 
> 2.10.2 and OSGi. I asked about 2.10.2 in a different thread
> 
> I am using Maven to build my Scala library but to resolve the bad POM
> definition issue, https://issues.apache.org/jira/browse/KAFKA-1064, I
> added all the required dependencies to my POM and thought I had to use
> Scala 2.9.2 but it seems to build just fine with 2.10.2, this should be
> fine, right?
> 
> However, I am getting java.lang.NoClassDefFoundError: 
> scala/reflect/ClassManifest error calling ProducerConfig
>  val config = new ProducerConfig(props)
> 
> I believe this is because Kafka is built with 2.9.2 so I made the following 
> changes to Build.scala
>  scalaVersion := “2.10.2” ,
>  publishTo := Some(Resolver.file("file",  new 
> File(Path.userHome.absolutePath+"/.m2/repository"))),
> 
> I should be able to run “sbt publish” to my local maven repository. However, 
> I also need to package this library in OSGi format since I am using the 
> library in OSGi, does anyone know if this works in OSGi? I would also need to 
> include addSbtPlugin("com.typesafe.sbt" % "sbt-osgi" % "0.4.0”) for OSGi 
> bundling.
> 
> Thanks for any help!
> Bob
> 
> 
> 


Re: Kafka 0.8, Scala 2.10.2, OSGi

2013-10-30 Thread Ngu, Bob
Thanks, I did just that, I found the kafka github repo, grabbed the 0.8
branch and it builds fine now. Any thoughts on the OSGi support question?

On 10/30/13, 4:00 PM, "Joe Stein"  wrote:

>Can you go off the 0.8 branch please? We're in the process of the 0.8.0
>release so if you still have problems that were not resolved we can know
>what they are.
>
>
>/***
> Joe Stein
> Founder, Principal Consultant
> Big Data Open Source Security LLC
> http://www.stealth.ly
> Twitter: @allthingshadoop
>/
>
>
>On Oct 30, 2013, at 6:26 PM, "Ngu, Bob"  wrote:
>
>> Hi,
>> 
>> I decided to start a new thread regarding Kafka 0.8 beta1 support for
>>Scala 2.10.2 and OSGi. I asked about 2.10.2 in a different thread
>> 
>> I am using Maven to build my Scala library but to resolve the bad POM
>> definition issue, https://issues.apache.org/jira/browse/KAFKA-1064, I
>> added all the required dependencies to my POM and thought I had to use
>> Scala 2.9.2 but it seems to build just fine with 2.10.2, this should be
>> fine, right?
>> 
>> However, I am getting java.lang.NoClassDefFoundError:
>>scala/reflect/ClassManifest error calling ProducerConfig
>>  val config = new ProducerConfig(props)
>> 
>> I believe this is because Kafka is built with 2.9.2 so I made the
>>following changes to Build.scala
>>  scalaVersion := ³2.10.2² ,
>>  publishTo := Some(Resolver.file("file",  new
>>File(Path.userHome.absolutePath+"/.m2/repository"))),
>> 
>> I should be able to run ³sbt publish² to my local maven repository.
>>However, I also need to package this library in OSGi format since I am
>>using the library in OSGi, does anyone know if this works in OSGi? I
>>would also need to include addSbtPlugin("com.typesafe.sbt" % "sbt-osgi"
>>% "0.4.0²) for OSGi bundling.
>> 
>> Thanks for any help!
>> Bob
>> 
>> 
>> 



Re: Kafka 0.8, Scala 2.10.2, OSGi

2013-10-30 Thread Ngu, Bob
Hi Joe,

I ran into a number of missing PGP key error with “sbt publish-local”
[error] (Kafka/*:packaged-artifacts) java.io.FileNotFoundException:
/Users/rngu/.sbt/gpg/secring.asc (No such file or directory)
[error] (perf/*:packaged-artifacts) java.io.FileNotFoundException:
/Users/rngu/.sbt/gpg/secring.asc (No such file or directory)
[error] (contrib/*:packaged-artifacts) java.io.FileNotFoundException:
/Users/rngu/.sbt/gpg/secring.asc (No such file or directory)
[error] (hadoop-consumer/*:packaged-artifacts)
java.io.FileNotFoundException: /Users/rngu/.sbt/gpg/secring.asc (No such
file or directory)
[error] (hadoop-producer/*:packaged-artifacts)
java.io.FileNotFoundException: /Users/rngu/.sbt/gpg/secring.asc (No such
file or directory)
[error] (java-examples/*:packaged-artifacts)
java.io.FileNotFoundException: /Users/rngu/.sbt/gpg/secring.asc (No such
file or directory)
[error] (core/*:packaged-artifacts) java.io.FileNotFoundException:
/Users/rngu/.sbt/gpg/secring.asc (No such file or directory)


Any ideas?
Thanks,
Bob

On 10/30/13, 4:12 PM, "Ngu, Bob"  wrote:

>Thanks, I did just that, I found the kafka github repo, grabbed the 0.8
>branch and it builds fine now. Any thoughts on the OSGi support question?
>
>On 10/30/13, 4:00 PM, "Joe Stein"  wrote:
>
>>Can you go off the 0.8 branch please? We're in the process of the 0.8.0
>>release so if you still have problems that were not resolved we can know
>>what they are.
>>
>>
>>/***
>> Joe Stein
>> Founder, Principal Consultant
>> Big Data Open Source Security LLC
>> http://www.stealth.ly
>> Twitter: @allthingshadoop
>>/
>>
>>
>>On Oct 30, 2013, at 6:26 PM, "Ngu, Bob"  wrote:
>>
>>> Hi,
>>> 
>>> I decided to start a new thread regarding Kafka 0.8 beta1 support for
>>>Scala 2.10.2 and OSGi. I asked about 2.10.2 in a different thread
>>> 
>>> I am using Maven to build my Scala library but to resolve the bad POM
>>> definition issue, https://issues.apache.org/jira/browse/KAFKA-1064, I
>>> added all the required dependencies to my POM and thought I had to use
>>> Scala 2.9.2 but it seems to build just fine with 2.10.2, this should be
>>> fine, right?
>>> 
>>> However, I am getting java.lang.NoClassDefFoundError:
>>>scala/reflect/ClassManifest error calling ProducerConfig
>>>  val config = new ProducerConfig(props)
>>> 
>>> I believe this is because Kafka is built with 2.9.2 so I made the
>>>following changes to Build.scala
>>>  scalaVersion := ³2.10.2² ,
>>>  publishTo := Some(Resolver.file("file",  new
>>>File(Path.userHome.absolutePath+"/.m2/repository"))),
>>> 
>>> I should be able to run ³sbt publish² to my local maven repository.
>>>However, I also need to package this library in OSGi format since I am
>>>using the library in OSGi, does anyone know if this works in OSGi? I
>>>would also need to include addSbtPlugin("com.typesafe.sbt" % "sbt-osgi"
>>>% "0.4.0²) for OSGi bundling.
>>> 
>>> Thanks for any help!
>>> Bob
>>> 
>>> 
>>> 
>



Re: Kafka 0.8, Scala 2.10.2, OSGi

2013-10-30 Thread Joe Stein
you need a pgp keypair.

if you use sbt you could try http://www.scala-sbt.org/sbt-pgp/usage.html

or without sbt get OpenPGP installed http://linux.die.net/man/1/gpg2 and
then

gpg2 --gen-key


On Wed, Oct 30, 2013 at 8:05 PM, Ngu, Bob  wrote:

> Hi Joe,
>
> I ran into a number of missing PGP key error with “sbt publish-local”
> [error] (Kafka/*:packaged-artifacts) java.io.FileNotFoundException:
> /Users/rngu/.sbt/gpg/secring.asc (No such file or directory)
> [error] (perf/*:packaged-artifacts) java.io.FileNotFoundException:
> /Users/rngu/.sbt/gpg/secring.asc (No such file or directory)
> [error] (contrib/*:packaged-artifacts) java.io.FileNotFoundException:
> /Users/rngu/.sbt/gpg/secring.asc (No such file or directory)
> [error] (hadoop-consumer/*:packaged-artifacts)
> java.io.FileNotFoundException: /Users/rngu/.sbt/gpg/secring.asc (No such
> file or directory)
> [error] (hadoop-producer/*:packaged-artifacts)
> java.io.FileNotFoundException: /Users/rngu/.sbt/gpg/secring.asc (No such
> file or directory)
> [error] (java-examples/*:packaged-artifacts)
> java.io.FileNotFoundException: /Users/rngu/.sbt/gpg/secring.asc (No such
> file or directory)
> [error] (core/*:packaged-artifacts) java.io.FileNotFoundException:
> /Users/rngu/.sbt/gpg/secring.asc (No such file or directory)
>
>
> Any ideas?
> Thanks,
> Bob
>
> On 10/30/13, 4:12 PM, "Ngu, Bob"  wrote:
>
> >Thanks, I did just that, I found the kafka github repo, grabbed the 0.8
> >branch and it builds fine now. Any thoughts on the OSGi support question?
> >
> >On 10/30/13, 4:00 PM, "Joe Stein"  wrote:
> >
> >>Can you go off the 0.8 branch please? We're in the process of the 0.8.0
> >>release so if you still have problems that were not resolved we can know
> >>what they are.
> >>
> >>
> >>/***
> >> Joe Stein
> >> Founder, Principal Consultant
> >> Big Data Open Source Security LLC
> >> http://www.stealth.ly
> >> Twitter: @allthingshadoop
> >>/
> >>
> >>
> >>On Oct 30, 2013, at 6:26 PM, "Ngu, Bob"  wrote:
> >>
> >>> Hi,
> >>>
> >>> I decided to start a new thread regarding Kafka 0.8 beta1 support for
> >>>Scala 2.10.2 and OSGi. I asked about 2.10.2 in a different thread
> >>>
> >>> I am using Maven to build my Scala library but to resolve the bad POM
> >>> definition issue, https://issues.apache.org/jira/browse/KAFKA-1064, I
> >>> added all the required dependencies to my POM and thought I had to use
> >>> Scala 2.9.2 but it seems to build just fine with 2.10.2, this should be
> >>> fine, right?
> >>>
> >>> However, I am getting java.lang.NoClassDefFoundError:
> >>>scala/reflect/ClassManifest error calling ProducerConfig
> >>>  val config = new ProducerConfig(props)
> >>>
> >>> I believe this is because Kafka is built with 2.9.2 so I made the
> >>>following changes to Build.scala
> >>>  scalaVersion := ³2.10.2² ,
> >>>  publishTo := Some(Resolver.file("file",  new
> >>>File(Path.userHome.absolutePath+"/.m2/repository"))),
> >>>
> >>> I should be able to run ³sbt publish² to my local maven repository.
> >>>However, I also need to package this library in OSGi format since I am
> >>>using the library in OSGi, does anyone know if this works in OSGi? I
> >>>would also need to include addSbtPlugin("com.typesafe.sbt" % "sbt-osgi"
> >>>% "0.4.0²) for OSGi bundling.
> >>>
> >>> Thanks for any help!
> >>> Bob
> >>>
> >>>
> >>>
> >
>
>


Re: Kafka 0.8, Scala 2.10.2, OSGi

2013-10-30 Thread Joe Stein
In regards to OSGi maybe it is something you would propose and help out
with for the client rewrite
https://cwiki.apache.org/confluence/display/KAFKA/Client+Rewrite I don't
know what is entailed for coding and long term support and what the
community benefit would be.  If you could help with any of that it would go
towards making it happening.

/***
 Joe Stein
 Founder, Principal Consultant
 Big Data Open Source Security LLC
 http://www.stealth.ly
 Twitter: @allthingshadoop 
/


On Wed, Oct 30, 2013 at 7:12 PM, Ngu, Bob  wrote:

> Thanks, I did just that, I found the kafka github repo, grabbed the 0.8
> branch and it builds fine now. Any thoughts on the OSGi support question?
>
> On 10/30/13, 4:00 PM, "Joe Stein"  wrote:
>
> >Can you go off the 0.8 branch please? We're in the process of the 0.8.0
> >release so if you still have problems that were not resolved we can know
> >what they are.
> >
> >
> >/***
> > Joe Stein
> > Founder, Principal Consultant
> > Big Data Open Source Security LLC
> > http://www.stealth.ly
> > Twitter: @allthingshadoop
> >/
> >
> >
> >On Oct 30, 2013, at 6:26 PM, "Ngu, Bob"  wrote:
> >
> >> Hi,
> >>
> >> I decided to start a new thread regarding Kafka 0.8 beta1 support for
> >>Scala 2.10.2 and OSGi. I asked about 2.10.2 in a different thread
> >>
> >> I am using Maven to build my Scala library but to resolve the bad POM
> >> definition issue, https://issues.apache.org/jira/browse/KAFKA-1064, I
> >> added all the required dependencies to my POM and thought I had to use
> >> Scala 2.9.2 but it seems to build just fine with 2.10.2, this should be
> >> fine, right?
> >>
> >> However, I am getting java.lang.NoClassDefFoundError:
> >>scala/reflect/ClassManifest error calling ProducerConfig
> >>  val config = new ProducerConfig(props)
> >>
> >> I believe this is because Kafka is built with 2.9.2 so I made the
> >>following changes to Build.scala
> >>  scalaVersion := ³2.10.2² ,
> >>  publishTo := Some(Resolver.file("file",  new
> >>File(Path.userHome.absolutePath+"/.m2/repository"))),
> >>
> >> I should be able to run ³sbt publish² to my local maven repository.
> >>However, I also need to package this library in OSGi format since I am
> >>using the library in OSGi, does anyone know if this works in OSGi? I
> >>would also need to include addSbtPlugin("com.typesafe.sbt" % "sbt-osgi"
> >>% "0.4.0²) for OSGi bundling.
> >>
> >> Thanks for any help!
> >> Bob
> >>
> >>
> >>
>
>


Re: Questions about producer API

2013-10-30 Thread Neha Narkhede
Agree that it is somewhat awkward to use zookeeper for broker discovery on
consumer, but a broker list on the producer. There were a couple of
discussions on the mailing list suggesting using zookeeper on the producer,
at least for discovering the brokers for the first time. However, we are
starting on the Client Rewrite project which is targeted for 0.9. That is
something we can consider changing on Kafka 0.9. If there is sufficient
interest, we can look making the zookeeper config change on the producer
soon. But that is something to discuss on a JIRA.

Thanks,
Neha


On Wed, Oct 30, 2013 at 9:53 AM, Roger Hoover wrote:

> Hi,
>
> I'm still getting started with Kafka and was curious why there is an
> asymmetry between the producer and consumer APIs.  Why does the producer
> config take a list of brokers where as the consumer config takes a list of
> brokers?
>
> Thanks,
>
> Roger
>


Roadmap

2013-10-30 Thread Richard Rodseth
Hi

I just started looking at Kafka today. Installing 0.8 Beta 1 went well and
I was able to run the command-line producer and consumer. When I tried to
write some Java code, I ran into problems with Maven dependencies (it was
looking for Scala.Product). I used the appropriate  from Maven
central.

Looking at the roadmap, 0.8 seems imminent, but no date is listed. 0.9 has
open issues but is listed as due tomorrow.

https://issues.apache.org/jira/browse/KAFKA?selectedTab=com.atlassian.jira.plugin.system.project%3Aroadmap-panel

What's the latest news, and will 0.8 resolve this issue?

Thanks! I'm excited about giving it a whirl.


Why Apache Kafka is bettern than any other Message System?

2013-10-30 Thread 이연옥
Hi, all.
I just got curiosity why Apache Kafka is better than any other Message System 
in terms of throughput, and durability.
What’s the fact to let Kafka have better performance?
I read some documents and researched, however, I am not quite sure about that..
And Zookeeper in Apache Kafka seems like a SPOF, Doesn’t it?
How Kafka be implemented in real world web-site?


Re: Why Apache Kafka is bettern than any other Message System?

2013-10-30 Thread Philip O'Toole
On Wed, Oct 30, 2013 at 8:13 PM, Lee, Yeon Ok (이연옥) wrote:

> Hi, all.
> I just got curiosity why Apache Kafka is better than any other Message
> System in terms of throughput, and durability.
>

Because it's brilliant, that's why. :-)


> What’s the fact to let Kafka have better performance?
>

Because it doesn't track consumer state, not in 0.72 anyway. And it makes
very efficient use of disks.


> I read some documents and researched, however, I am not quite sure about
> that..
> And Zookeeper in Apache Kafka seems like a SPOF, Doesn’t it?
>

Kafka can be configured to run without Zookeeper, if you are that
concerned, at the cost of more rigid system.


> How Kafka be implemented in real world web-site?
>

Depends what you want to do.  Whether you need to use it is the real
question.

Philip


Connection reset by peer Error in Kafka server.log

2013-10-30 Thread Banerjee, Aparup
Hi,

I keep getting this error in Kafka server.log. I don't see anything in my 
producer or consumer log. Any idea what can be going on here?

Aparup

2013-10-31 02:37:13,917] INFO Closing socket connection to /x.x.x.x. 
(kafka.network.Processor)
[2013-10-31 02:37:21,645] INFO Closing socket connection to /x.x.x.x. 
(kafka.network.Processor)
[2013-10-31 02:37:21,713] ERROR Closing socket for /x.x.x.x because of error 
(kafka.network.Processor)
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:197)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379)
at kafka.utils.Utils$.read(Utils.scala:394)
at 
kafka.network.BoundedByteBufferReceive.readFrom(BoundedByteBufferReceive.scala:54)
at kafka.network.Processor.read(SocketServer.scala:339)
at kafka.network.Processor.run(SocketServer.scala:245)
at java.lang.Thread.run(Thread.java:744)



Re: Why Apache Kafka is bettern than any other Message System?

2013-10-30 Thread Neha Narkhede
You can find Kafka use cases described here -
http://kafka.apache.org/documentation.html#uses

Also, the design document is a good read to understand why Kafka is more
efficient - http://kafka.apache.org/documentation.html#design

Thanks,
Neha


On Wed, Oct 30, 2013 at 8:24 PM, Philip O'Toole  wrote:

> On Wed, Oct 30, 2013 at 8:13 PM, Lee, Yeon Ok (이연옥)  >wrote:
>
> > Hi, all.
> > I just got curiosity why Apache Kafka is better than any other Message
> > System in terms of throughput, and durability.
> >
>
> Because it's brilliant, that's why. :-)
>
>
> > What’s the fact to let Kafka have better performance?
> >
>
> Because it doesn't track consumer state, not in 0.72 anyway. And it makes
> very efficient use of disks.
>
>
> > I read some documents and researched, however, I am not quite sure about
> > that..
> > And Zookeeper in Apache Kafka seems like a SPOF, Doesn’t it?
> >
>
> Kafka can be configured to run without Zookeeper, if you are that
> concerned, at the cost of more rigid system.
>
>
> > How Kafka be implemented in real world web-site?
> >
>
> Depends what you want to do.  Whether you need to use it is the real
> question.
>
> Philip
>