Release versioning

2013-12-10 Thread David Arthur

Just a quick question:

0.8.0 was just released and the next planned release is 0.8.1, and the 
next major set of updates (coordinator, etc) is targeted at 0.9.0. It 
seems like we are using the pattern 0.. instead of 
..


So what happens if we have a bugfix release for 0.8.0? Would it bump to 
0.8.0.1?


A good example would be how to fix 
https://issues.apache.org/jira/browse/KAFKA-1174 - the solution probably 
involves bumping the version number.


Cheers
-David


Re: Release versioning

2013-12-10 Thread Jun Rao
Yes, we can do an 0.8.0.1 if there are critical fixes.

Thanks,

Jun


On Tue, Dec 10, 2013 at 5:52 AM, David Arthur  wrote:

> Just a quick question:
>
> 0.8.0 was just released and the next planned release is 0.8.1, and the
> next major set of updates (coordinator, etc) is targeted at 0.9.0. It seems
> like we are using the pattern 0.. instead of
> ..
>
> So what happens if we have a bugfix release for 0.8.0? Would it bump to
> 0.8.0.1?
>
> A good example would be how to fix https://issues.apache.org/
> jira/browse/KAFKA-1174 - the solution probably involves bumping the
> version number.
>
> Cheers
> -David
>


[jira] [Commented] (KAFKA-1177) DeleteTopics gives Successful message even if the specified Topic is not present

2013-12-10 Thread Jun Rao (JIRA)

[ 
https://issues.apache.org/jira/browse/KAFKA-1177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13844378#comment-13844378
 ] 

Jun Rao commented on KAFKA-1177:


Deleting topics is not supported yet. We have a jira (KAFKA-330) to track that.

> DeleteTopics gives Successful message even if the specified Topic is not 
> present
> 
>
> Key: KAFKA-1177
> URL: https://issues.apache.org/jira/browse/KAFKA-1177
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.8.0
> Environment: Centos 6.3 with jdk 1.6 and apache kafka 0.8
>Reporter: Siddhesh Sundar Toraskar
>Priority: Minor
>
> I am implementating basic functionalities of kafka. I have created some java 
> classes for creating topics , a producer to send messages,a ConsumerGroup to 
> consume those messages and at last,a class for deleting topic. My other 
> functionalities are working normal but my class for deleting topic is not 
> working as required. The problems are:
> 1. It is displaying 'deletion successful' for the topics not present.
> 2. If a topic is deleted and then another topic with same name is created, 
> the messages from previous deleted topics are consumed.(This happens till I 
> dont restart the server).
>  So, is it required that everytime I delete a topic, I have to restart my 
> kafka server?.Or, there are some other solutions?



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)


Re: Review Request 16092: Patch for KAFKA-1147

2013-12-10 Thread Jun Rao

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/16092/#review30101
---


It seems that in the consumer, we have a separate request timeout and socket 
timeout. The producer only has a request timeout now. Perhaps we should add the 
socket timeout in the producer too?

- Jun Rao


On Dec. 9, 2013, 5:14 p.m., Guozhang Wang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/16092/
> ---
> 
> (Updated Dec. 9, 2013, 5:14 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1147
> https://issues.apache.org/jira/browse/KAFKA-1147
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> KAFKA-1147.v2.1
> 
> 
> KAFKA-1147.v2
> 
> 
> Diffs
> -
> 
>   core/src/main/scala/kafka/consumer/ConsumerConfig.scala 
> c8c4212f62dd52158e6504cba87b300e6830e5fe 
>   core/src/main/scala/kafka/server/KafkaConfig.scala 
> a7e5b739454aec796070d95b003f20f79c84ef89 
> 
> Diff: https://reviews.apache.org/r/16092/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Guozhang Wang
> 
>



[jira] [Commented] (KAFKA-1171) Gradle build for Kafka

2013-12-10 Thread Jun Rao (JIRA)

[ 
https://issues.apache.org/jira/browse/KAFKA-1171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13844396#comment-13844396
 ] 

Jun Rao commented on KAFKA-1171:


David,

Thanks for the patch. Is that all we need to do to switch to Gradle? Does the 
patch support maven publishing and ide project build?

> Gradle build for Kafka
> --
>
> Key: KAFKA-1171
> URL: https://issues.apache.org/jira/browse/KAFKA-1171
> Project: Kafka
>  Issue Type: Improvement
>  Components: packaging
>Affects Versions: 0.8.1, 0.9.0
>Reporter: David Arthur
>Assignee: David Arthur
> Attachments: 0001-Adding-basic-Gradle-build.patch
>
>
> We have previously discussed moving away from SBT to an 
> easier-to-comprehend-and-debug build system such as Ant or Gradle. I put up a 
> patch for an Ant+Ivy build a while ago[1], and it sounded like people wanted 
> to check out Gradle as well.
> 1. https://issues.apache.org/jira/browse/KAFKA-855



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)


[jira] [Commented] (KAFKA-1171) Gradle build for Kafka

2013-12-10 Thread David Arthur (JIRA)

[ 
https://issues.apache.org/jira/browse/KAFKA-1171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13844401#comment-13844401
 ] 

David Arthur commented on KAFKA-1171:
-

Jun,

Gradle does have plugins for IDEs and Maven publishing (as well as many 
others). What it's missing right now is support for building against 
different Scala versions.




> Gradle build for Kafka
> --
>
> Key: KAFKA-1171
> URL: https://issues.apache.org/jira/browse/KAFKA-1171
> Project: Kafka
>  Issue Type: Improvement
>  Components: packaging
>Affects Versions: 0.8.1, 0.9.0
>Reporter: David Arthur
>Assignee: David Arthur
> Attachments: 0001-Adding-basic-Gradle-build.patch
>
>
> We have previously discussed moving away from SBT to an 
> easier-to-comprehend-and-debug build system such as Ant or Gradle. I put up a 
> patch for an Ant+Ivy build a while ago[1], and it sounded like people wanted 
> to check out Gradle as well.
> 1. https://issues.apache.org/jira/browse/KAFKA-855



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)


[jira] [Updated] (KAFKA-1079) Liars in PrimitiveApiTest that promise to test api in compression mode, but don't do this actually

2013-12-10 Thread Kostya Golikov (JIRA)

 [ 
https://issues.apache.org/jira/browse/KAFKA-1079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kostya Golikov updated KAFKA-1079:
--

Fix Version/s: 0.8.1
   Status: Patch Available  (was: Open)

>From 7845a5af42ee44f9786542178c0789d93c66429f Mon Sep 17 00:00:00 2001
From: Kostya Golikov 
Date: Thu, 24 Oct 2013 00:48:56 +0400
Subject: [PATCH] Fixed liars in primary api test: now checking is done not
 only against no-compression producer, but against with-compression producer
 as well.

---
 .../unit/kafka/integration/PrimitiveApiTest.scala  | 141 ++---
 1 file changed, 40 insertions(+), 101 deletions(-)

diff --git a/core/src/test/scala/unit/kafka/integration/PrimitiveApiTest.scala 
b/core/src/test/scala/unit/kafka/integration/PrimitiveApiTest.scala
index 5f331d2..c001b4e 100644
--- a/core/src/test/scala/unit/kafka/integration/PrimitiveApiTest.scala
+++ b/core/src/test/scala/unit/kafka/integration/PrimitiveApiTest.scala
@@ -35,12 +35,12 @@ import kafka.utils.{TestUtils, Utils}
  * End to end tests of the primitive apis against a local server
  */
 class PrimitiveApiTest extends JUnit3Suite with ProducerConsumerTestHarness 
with ZooKeeperTestHarness {
+  val requestHandlerLogger = Logger.getLogger(classOf[KafkaRequestHandler])
 
-  val port = TestUtils.choosePort
+  val port = TestUtils.choosePort()
   val props = TestUtils.createBrokerConfig(0, port)
   val config = new KafkaConfig(props)
   val configs = List(config)
-  val requestHandlerLogger = Logger.getLogger(classOf[KafkaRequestHandler])
 
   def testFetchRequestCanProperlySerialize() {
 val request = new FetchRequestBuilder()
@@ -100,7 +100,7 @@ class PrimitiveApiTest extends JUnit3Suite with 
ProducerConsumerTestHarness with
 val stringProducer1 = new Producer[String, String](config)
 stringProducer1.send(new KeyedMessage[String, String](topic, 
"test-message"))
 
-var fetched = consumer.fetch(new FetchRequestBuilder().addFetch(topic, 0, 
0, 1).build())
+val fetched = consumer.fetch(new FetchRequestBuilder().addFetch(topic, 0, 
0, 1).build())
 val messageSet = fetched.messageSet(topic, 0)
 assertTrue(messageSet.iterator.hasNext)
 
@@ -108,8 +108,8 @@ class PrimitiveApiTest extends JUnit3Suite with 
ProducerConsumerTestHarness with
 assertEquals("test-message", 
Utils.readString(fetchedMessageAndOffset.message.payload, "UTF-8"))
   }
 
-  def testProduceAndMultiFetch() {
-createSimpleTopicsAndAwaitLeader(zkClient, List("test1", "test2", "test3", 
"test4"), config.brokerId)
+  private def produceAndMultiFetch(producer: Producer[String, String]) {
+createSimpleTopicsAndAwaitLeader(zkClient, List("test1", "test2", "test3", 
"test4"))
 
 // send some messages
 val topics = List(("test4", 0), ("test1", 0), ("test2", 0), ("test3", 0));
@@ -171,117 +171,56 @@ class PrimitiveApiTest extends JUnit3Suite with 
ProducerConsumerTestHarness with
 requestHandlerLogger.setLevel(Level.ERROR)
   }
 
-  def testProduceAndMultiFetchWithCompression() {
-createSimpleTopicsAndAwaitLeader(zkClient, List("test1", "test2", "test3", 
"test4"), config.brokerId)
-
-// send some messages
-val topics = List(("test4", 0), ("test1", 0), ("test2", 0), ("test3", 0));
-{
-  val messages = new mutable.HashMap[String, Seq[String]]
-  val builder = new FetchRequestBuilder()
-  for( (topic, partition) <- topics) {
-val messageList = List("a_" + topic, "b_" + topic)
-val producerData = messageList.map(new KeyedMessage[String, 
String](topic, topic, _))
-messages += topic -> messageList
-producer.send(producerData:_*)
-builder.addFetch(topic, partition, 0, 1)
-  }
-
-  // wait a bit for produced message to be available
-  val request = builder.build()
-  val response = consumer.fetch(request)
-  for( (topic, partition) <- topics) {
-val fetched = response.messageSet(topic, partition)
-assertEquals(messages(topic), fetched.map(messageAndOffset => 
Utils.readString(messageAndOffset.message.payload)))
-  }
-}
-
-// temporarily set request handler logger to a higher level
-requestHandlerLogger.setLevel(Level.FATAL)
-
-{
-  // send some invalid offsets
-  val builder = new FetchRequestBuilder()
-  for( (topic, partition) <- topics)
-builder.addFetch(topic, partition, -1, 1)
-
-  try {
-val request = builder.build()
-val response = consumer.fetch(request)
-response.data.values.foreach(pdata => 
ErrorMapping.maybeThrowException(pdata.error))
-fail("Expected exception when fetching message with invalid offset")
-  } catch {
-case e: OffsetOutOfRangeException => "this is good"
-  }
-}
-
-{
-  // send some invalid partitions
-  val builder = new FetchRequestBuilder()
-  for( (topic, _) <- topics)
-builder.addFetch(topic, 

Re: Review Request 16092: Patch for KAFKA-1147

2013-12-10 Thread Guozhang Wang


> On Dec. 10, 2013, 4:14 p.m., Jun Rao wrote:
> > It seems that in the consumer, we have a separate request timeout and 
> > socket timeout. The producer only has a request timeout now. Perhaps we 
> > should add the socket timeout in the producer too?

Currently request.timeout.ms of producer config is used as both socket timeout 
for the channel and produce ack timeout. Are you suggesting decoupling them by 
adding another socket timeout parameter?


- Guozhang


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/16092/#review30101
---


On Dec. 9, 2013, 5:14 p.m., Guozhang Wang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/16092/
> ---
> 
> (Updated Dec. 9, 2013, 5:14 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1147
> https://issues.apache.org/jira/browse/KAFKA-1147
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> KAFKA-1147.v2.1
> 
> 
> KAFKA-1147.v2
> 
> 
> Diffs
> -
> 
>   core/src/main/scala/kafka/consumer/ConsumerConfig.scala 
> c8c4212f62dd52158e6504cba87b300e6830e5fe 
>   core/src/main/scala/kafka/server/KafkaConfig.scala 
> a7e5b739454aec796070d95b003f20f79c84ef89 
> 
> Diff: https://reviews.apache.org/r/16092/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Guozhang Wang
> 
>



[jira] [Commented] (KAFKA-1174) Empty jar in Maven Central for Scala 2.8.0

2013-12-10 Thread Joe Stein (JIRA)

[ 
https://issues.apache.org/jira/browse/KAFKA-1174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13844546#comment-13844546
 ] 

Joe Stein commented on KAFKA-1174:
--

The issue looks like it was in the packing and publish for the release.

I staged a 2.8.0 jar (with only changing the versions)

https://repository.apache.org/content/groups/staging/


  org.apache.kafka
  kafka_2.8.0
  0.8.0-s28fixTest


lets verify that this solves all the issues before deciding next steps

> Empty jar in Maven Central for Scala 2.8.0
> --
>
> Key: KAFKA-1174
> URL: https://issues.apache.org/jira/browse/KAFKA-1174
> Project: Kafka
>  Issue Type: Bug
>  Components: packaging
>Affects Versions: 0.8.0
>Reporter: David Arthur
>Priority: Critical
>
> As reported by wildag on IRC
> In Maven Central, the jar for kafka core only contains the license and notice 
> files, no classes. I checked the other Scala versions and they seem fine.
> See: 
> http://search.maven.org/#artifactdetails%7Corg.apache.kafka%7Ckafka_2.8.0%7C0.8.0%7Cjar
> Unless we can invoke the power of the Sonatype gods, I think we must bump the 
> version number to fix this (or wait until 0.8.1 drops).



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)


Re: Release versioning

2013-12-10 Thread Joe Stein
I published a new JAR to staging repo to confirm that the issue was
originally just packaging and publishing issue and not code related.

Maven central is immutable so if we pushed 0.8.0 (without changing any
version or code) it would only show up in the apache release repo.

if we change the version we could just vote and publish the 2.8.0 jar and
maybe name the version to be related to fixing that as 0.8.0.1 to me means
there is a bug/defect in the 0.8.0 release from a code perspective and that
is not the case (assuming re-publishing 2.8.0 jar fixes everything) ...
maybe 0.8.0-scala280 or something... (don't lave that but it feels more
meaningful to me)

can someone verify that the new 2.8.0 staging jar is working now?

/***
 Joe Stein
 Founder, Principal Consultant
 Big Data Open Source Security LLC
 http://www.stealth.ly
 Twitter: @allthingshadoop 
/


On Tue, Dec 10, 2013 at 11:04 AM, Jun Rao  wrote:

> Yes, we can do an 0.8.0.1 if there are critical fixes.
>
> Thanks,
>
> Jun
>
>
> On Tue, Dec 10, 2013 at 5:52 AM, David Arthur  wrote:
>
> > Just a quick question:
> >
> > 0.8.0 was just released and the next planned release is 0.8.1, and the
> > next major set of updates (coordinator, etc) is targeted at 0.9.0. It
> seems
> > like we are using the pattern 0.. instead of
> > ..
> >
> > So what happens if we have a bugfix release for 0.8.0? Would it bump to
> > 0.8.0.1?
> >
> > A good example would be how to fix https://issues.apache.org/
> > jira/browse/KAFKA-1174 - the solution probably involves bumping the
> > version number.
> >
> > Cheers
> > -David
> >
>


[jira] [Commented] (KAFKA-1174) Empty jar in Maven Central for Scala 2.8.0

2013-12-10 Thread David Arthur (JIRA)

[ 
https://issues.apache.org/jira/browse/KAFKA-1174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13844568#comment-13844568
 ] 

David Arthur commented on KAFKA-1174:
-

+1 verified:
* Jars in 
https://repository.apache.org/content/groups/staging/org/apache/kafka/kafka_2.8.0/0.8.0/
 are emtpy (only a few Kb)
* Jars in 
https://repository.apache.org/content/groups/staging/org/apache/kafka/kafka_2.8.0/0.8.0-s28fixTest/
 are "real" (a few Mb).

I don't suppose it's possible to replace the Jars on Maven Central? If not, I 
guess the right thing to do is do another release and bump the version to 
0.8.0.1. Either that, or we create a special version for the 2.8.0 artifact 
like "0.8.0-fixed" or something.

> Empty jar in Maven Central for Scala 2.8.0
> --
>
> Key: KAFKA-1174
> URL: https://issues.apache.org/jira/browse/KAFKA-1174
> Project: Kafka
>  Issue Type: Bug
>  Components: packaging
>Affects Versions: 0.8.0
>Reporter: David Arthur
>Priority: Critical
>
> As reported by wildag on IRC
> In Maven Central, the jar for kafka core only contains the license and notice 
> files, no classes. I checked the other Scala versions and they seem fine.
> See: 
> http://search.maven.org/#artifactdetails%7Corg.apache.kafka%7Ckafka_2.8.0%7C0.8.0%7Cjar
> Unless we can invoke the power of the Sonatype gods, I think we must bump the 
> version number to fix this (or wait until 0.8.1 drops).



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)


[jira] [Updated] (KAFKA-1147) Consumer socket timeout should be greater than fetch max wait

2013-12-10 Thread Guozhang Wang (JIRA)

 [ 
https://issues.apache.org/jira/browse/KAFKA-1147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang updated KAFKA-1147:
-

Attachment: KAFKA-1147_2013-12-10_14:31:46.patch

> Consumer socket timeout should be greater than fetch max wait
> -
>
> Key: KAFKA-1147
> URL: https://issues.apache.org/jira/browse/KAFKA-1147
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.0, 0.8.1
>Reporter: Joel Koshy
>Assignee: Guozhang Wang
> Fix For: 0.8.1
>
> Attachments: KAFKA-1147.patch, KAFKA-1147_2013-12-07_18:22:18.patch, 
> KAFKA-1147_2013-12-09_09:14:24.patch, KAFKA-1147_2013-12-10_14:31:46.patch
>
>
> From the mailing list:
> The consumer-config documentation states that "The actual timeout set
> will be max.fetch.wait + socket.timeout.ms." - however, that change
> seems to have been lost in the code a while ago - we should either fix the 
> doc or re-introduce the addition.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)


Re: Review Request 16092: Patch for KAFKA-1147

2013-12-10 Thread Guozhang Wang

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/16092/
---

(Updated Dec. 10, 2013, 10:31 p.m.)


Review request for kafka.


Bugs: KAFKA-1147
https://issues.apache.org/jira/browse/KAFKA-1147


Repository: kafka


Description (updated)
---

KAFKA-1147.v2.2


KAFKA-1147.v2.1


KAFKA-1147.v2


Diffs (updated)
-

  core/src/main/scala/kafka/consumer/ConsumerConfig.scala 
c8c4212f62dd52158e6504cba87b300e6830e5fe 
  core/src/main/scala/kafka/producer/SyncProducer.scala 
419156eb143fb04a305c91c964307a89ba5a82fa 
  core/src/main/scala/kafka/producer/SyncProducerConfig.scala 
69b2d0c11bb1412ce76d566f285333c806be301a 
  core/src/main/scala/kafka/server/KafkaConfig.scala 
a7e5b739454aec796070d95b003f20f79c84ef89 

Diff: https://reviews.apache.org/r/16092/diff/


Testing
---


Thanks,

Guozhang Wang



[jira] [Commented] (KAFKA-1147) Consumer socket timeout should be greater than fetch max wait

2013-12-10 Thread Guozhang Wang (JIRA)

[ 
https://issues.apache.org/jira/browse/KAFKA-1147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13844758#comment-13844758
 ] 

Guozhang Wang commented on KAFKA-1147:
--

Updated reviewboard https://reviews.apache.org/r/16092/
 against branch origin/trunk

> Consumer socket timeout should be greater than fetch max wait
> -
>
> Key: KAFKA-1147
> URL: https://issues.apache.org/jira/browse/KAFKA-1147
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.0, 0.8.1
>Reporter: Joel Koshy
>Assignee: Guozhang Wang
> Fix For: 0.8.1
>
> Attachments: KAFKA-1147.patch, KAFKA-1147_2013-12-07_18:22:18.patch, 
> KAFKA-1147_2013-12-09_09:14:24.patch, KAFKA-1147_2013-12-10_14:31:46.patch
>
>
> From the mailing list:
> The consumer-config documentation states that "The actual timeout set
> will be max.fetch.wait + socket.timeout.ms." - however, that change
> seems to have been lost in the code a while ago - we should either fix the 
> doc or re-introduce the addition.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)


[jira] [Commented] (KAFKA-1171) Gradle build for Kafka

2013-12-10 Thread Chris Riccomini (JIRA)

[ 
https://issues.apache.org/jira/browse/KAFKA-1171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13844763#comment-13844763
 ] 

Chris Riccomini commented on KAFKA-1171:


For cross-building Scala versions, have a look at:

https://issues.apache.org/jira/browse/SAMZA-34

[~szczepiq] provided the example zip file.

> Gradle build for Kafka
> --
>
> Key: KAFKA-1171
> URL: https://issues.apache.org/jira/browse/KAFKA-1171
> Project: Kafka
>  Issue Type: Improvement
>  Components: packaging
>Affects Versions: 0.8.1, 0.9.0
>Reporter: David Arthur
>Assignee: David Arthur
> Attachments: 0001-Adding-basic-Gradle-build.patch
>
>
> We have previously discussed moving away from SBT to an 
> easier-to-comprehend-and-debug build system such as Ant or Gradle. I put up a 
> patch for an Ant+Ivy build a while ago[1], and it sounded like people wanted 
> to check out Gradle as well.
> 1. https://issues.apache.org/jira/browse/KAFKA-855



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)


[jira] [Commented] (KAFKA-1171) Gradle build for Kafka

2013-12-10 Thread Chris Riccomini (JIRA)

[ 
https://issues.apache.org/jira/browse/KAFKA-1171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13844780#comment-13844780
 ] 

Chris Riccomini commented on KAFKA-1171:


For the record, a good set of examples on how to build OSS multi-module Gradle 
projects is Netflix's GitHub account:

  https://github.com/Netflix

Their account contains a number of projects that use Gradle (Lipstick, 
astyanax, servo, etc). In addition, they have a stub repo for bootstrapping new 
projects with Gradle.

  https://github.com/Netflix/gradle-template

> Gradle build for Kafka
> --
>
> Key: KAFKA-1171
> URL: https://issues.apache.org/jira/browse/KAFKA-1171
> Project: Kafka
>  Issue Type: Improvement
>  Components: packaging
>Affects Versions: 0.8.1, 0.9.0
>Reporter: David Arthur
>Assignee: David Arthur
> Attachments: 0001-Adding-basic-Gradle-build.patch
>
>
> We have previously discussed moving away from SBT to an 
> easier-to-comprehend-and-debug build system such as Ant or Gradle. I put up a 
> patch for an Ant+Ivy build a while ago[1], and it sounded like people wanted 
> to check out Gradle as well.
> 1. https://issues.apache.org/jira/browse/KAFKA-855



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)


[jira] [Commented] (KAFKA-1171) Gradle build for Kafka

2013-12-10 Thread David Arthur (JIRA)

[ 
https://issues.apache.org/jira/browse/KAFKA-1171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13844789#comment-13844789
 ] 

David Arthur commented on KAFKA-1171:
-

[~criccomini] thanks for the pointer.

When I try out that sample you have on the SAMZA issue, I get the following:

{code}

FAILURE: Build failed with an exception.

* Where:
Script '/Users/mumrah/Downloads/multi-scala/scala.gradle' line: 44

* What went wrong:
A problem occurred evaluating root project 'multi-scala'.
> Failed to notify action.
   > groovy.lang.MissingMethodException: No signature of method: 
org.gradle.api.internal.artifacts.configurations.DefaultResolutionStrategy.eachDependency()
 is applicable for argument types: 
(scala_319e62rbueqtrto35ji6mp2hbc$_configureResolution_closure3) values: 
[scala_319e62rbueqtrto35ji6mp2hbc$_configureResolution_closure3@17e06b12]

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug 
option to get more log output.

BUILD FAILED

Total time: 4.169 secs
{code}

> Gradle build for Kafka
> --
>
> Key: KAFKA-1171
> URL: https://issues.apache.org/jira/browse/KAFKA-1171
> Project: Kafka
>  Issue Type: Improvement
>  Components: packaging
>Affects Versions: 0.8.1, 0.9.0
>Reporter: David Arthur
>Assignee: David Arthur
> Attachments: 0001-Adding-basic-Gradle-build.patch
>
>
> We have previously discussed moving away from SBT to an 
> easier-to-comprehend-and-debug build system such as Ant or Gradle. I put up a 
> patch for an Ant+Ivy build a while ago[1], and it sounded like people wanted 
> to check out Gradle as well.
> 1. https://issues.apache.org/jira/browse/KAFKA-855



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)


[jira] [Commented] (KAFKA-1171) Gradle build for Kafka

2013-12-10 Thread Chris Riccomini (JIRA)

[ 
https://issues.apache.org/jira/browse/KAFKA-1171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13844808#comment-13844808
 ] 

Chris Riccomini commented on KAFKA-1171:


That sounds a lot like a version incompatibility between the version of Gradle 
you're using, and what the project was built against (it includes no gradlew 
script).

I was able to successfully compile it use Samza's gradlew script. Might setup a 
gradlew in the project, and give that a shot.

> Gradle build for Kafka
> --
>
> Key: KAFKA-1171
> URL: https://issues.apache.org/jira/browse/KAFKA-1171
> Project: Kafka
>  Issue Type: Improvement
>  Components: packaging
>Affects Versions: 0.8.1, 0.9.0
>Reporter: David Arthur
>Assignee: David Arthur
> Attachments: 0001-Adding-basic-Gradle-build.patch
>
>
> We have previously discussed moving away from SBT to an 
> easier-to-comprehend-and-debug build system such as Ant or Gradle. I put up a 
> patch for an Ant+Ivy build a while ago[1], and it sounded like people wanted 
> to check out Gradle as well.
> 1. https://issues.apache.org/jira/browse/KAFKA-855



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)


Re: Review Request 15711: Patch for KAFKA-930

2013-12-10 Thread Guozhang Wang

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/15711/#review30149
---

Ship it!



core/src/main/scala/kafka/controller/KafkaController.scala


We can delay the creation of autoRebalanceScheduler here, setting it as var 
and initialize to null first. Hence in shutdown we can do 
if(autoRebalanceScheduler != null) {shutdown and set to null}


- Guozhang Wang


On Dec. 10, 2013, 6:52 a.m., Sriram Subramanian wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/15711/
> ---
> 
> (Updated Dec. 10, 2013, 6:52 a.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-930
> https://issues.apache.org/jira/browse/KAFKA-930
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> Address code review feedbacks
> 
> 
> Merge branch 'trunk' of http://git-wip-us.apache.org/repos/asf/kafka into 
> trunk
> 
> 
> commit missing code
> 
> 
> some more changes
> 
> 
> fix merge conflicts
> 
> 
> Add auto leader rebalance support
> 
> 
> Merge branch 'trunk' of http://git-wip-us.apache.org/repos/asf/kafka into 
> trunk
> 
> 
> Merge branch 'trunk' of http://git-wip-us.apache.org/repos/asf/kafka into 
> trunk
> 
> Conflicts:
>   core/src/main/scala/kafka/admin/AdminUtils.scala
>   core/src/main/scala/kafka/admin/TopicCommand.scala
> 
> change comments
> 
> 
> commit the remaining changes
> 
> 
> Move AddPartitions into TopicCommand
> 
> 
> Diffs
> -
> 
>   core/src/main/scala/kafka/controller/KafkaController.scala 
> 4c319aba97655e7c4ec97fac2e34de4e28c9f5d3 
>   core/src/main/scala/kafka/server/KafkaConfig.scala 
> b324344d0a383398db8bfe2cbeec2c1378fe13c9 
> 
> Diff: https://reviews.apache.org/r/15711/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sriram Subramanian
> 
>



Re: Review Request 15953: Patch for KAFKA-1134

2013-12-10 Thread Neha Narkhede

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/15953/#review30152
---

Ship it!


Ship It!

- Neha Narkhede


On Dec. 5, 2013, 7:13 p.m., Guozhang Wang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/15953/
> ---
> 
> (Updated Dec. 5, 2013, 7:13 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1134
> https://issues.apache.org/jira/browse/KAFKA-1134
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> KAFKA-1134.v1
> 
> 
> Diffs
> -
> 
>   core/src/main/scala/kafka/controller/ControllerChannelManager.scala 
> beca460dfe0f4df5ccd7f6358e44cbe742d256e5 
>   core/src/main/scala/kafka/controller/KafkaController.scala 
> 3beaf75f8285c8b6146aced2fefda4234cf1d307 
>   core/src/main/scala/kafka/controller/PartitionStateMachine.scala 
> 829163afab99706f7a34408eda2504c8262e572e 
> 
> Diff: https://reviews.apache.org/r/15953/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Guozhang Wang
> 
>



[jira] [Created] (KAFKA-1178) Replica fetcher thread dies while becoming follower

2013-12-10 Thread Neha Narkhede (JIRA)
Neha Narkhede created KAFKA-1178:


 Summary: Replica fetcher thread dies while becoming follower
 Key: KAFKA-1178
 URL: https://issues.apache.org/jira/browse/KAFKA-1178
 Project: Kafka
  Issue Type: Bug
  Components: replication
Affects Versions: 0.8.1
Reporter: Neha Narkhede
Assignee: Neha Narkhede
Priority: Critical


Replica fetcher thread dies due to 

2013/12/10 17:48:24.845 [ReplicaFetcherThread] [ReplicaFetcherThread--3-500], 
Error due to 
kafka.common.KafkaException: error processing data for partition 
[pts-test_foo,1] offset 340
at 
kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$1$$anonfun$apply$mcV$sp$2.apply(AbstractFetcherThread.scala:139)
at 
kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$1$$anonfun$apply$mcV$sp$2.apply(AbstractFetcherThread.scala:111)
at 
scala.collection.immutable.HashMap$HashMap1.foreach(HashMap.scala:125)
at 
scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:344)
at 
scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:344)
at 
kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$1.apply$mcV$sp(AbstractFetcherThread.scala:111)
at 
kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$1.apply(AbstractFetcherThread.scala:111)
at 
kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$1.apply(AbstractFetcherThread.scala:111)
at kafka.utils.Utils$.inLock(Utils.scala:538)
at 
kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:110)
at 
kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:88)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51)
Caused by: java.util.NoSuchElementException: None.get
at scala.None$.get(Option.scala:185)
at scala.None$.get(Option.scala:183)
at 
kafka.server.ReplicaFetcherThread.processPartitionData(ReplicaFetcherThread.scala:45)
at 
kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$1$$anonfun$apply$mcV$sp$2.apply(AbstractFetcherThread.scala:130)
... 11 more




--
This message was sent by Atlassian JIRA
(v6.1.4#6159)


[jira] [Commented] (KAFKA-1079) Liars in PrimitiveApiTest that promise to test api in compression mode, but don't do this actually

2013-12-10 Thread Jun Rao (JIRA)

[ 
https://issues.apache.org/jira/browse/KAFKA-1079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13845073#comment-13845073
 ] 

Jun Rao commented on KAFKA-1079:


There seems to be some weird characters in the pasted text. Could you attach 
the patch to this jira? Thanks,

> Liars in PrimitiveApiTest that promise to test api in compression mode, but 
> don't do this actually
> --
>
> Key: KAFKA-1079
> URL: https://issues.apache.org/jira/browse/KAFKA-1079
> Project: Kafka
>  Issue Type: Test
>  Components: core
>Affects Versions: 0.8.0
>Reporter: Kostya Golikov
>Priority: Minor
>  Labels: newbie, test
> Fix For: 0.8.1
>
> Attachments: testing-with-compression-producer.patch
>
>
> Long time ago (0.7) we had ByteBufferMessageSet as a part of api and it's 
> allowed us to control compression. Times goes on and now PrimitiveApiTest 
> have methods that promise to test api with compression enabled, but in fact 
> they don't. Moreover this methods almost entirely copy their counterparts 
> without compression. In particular I'm talking about 
> `testProduceAndMultiFetch` / `testProduceAndMultiFetchWithCompression` and 
> `testMultiProduce`/`testMultiProduceWithCompression` pairs. 
> The fix could be super-easy and soundness -- just parameterize methods with 
> producer of each type (with/without compression). Sadly but it isn't feasible 
> for junit3, so straightforward solution is to do the same ugly thing as 
> `testDefaultEncoderProducerAndFetchWithCompression` method does -- forget 
> about class-wide producer and roll-out it's own. I will attach path if that 
> is a problem indeed. 



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)


Review Request 16175: Patch for KAFKA-1178

2013-12-10 Thread Neha Narkhede

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/16175/
---

Review request for kafka.


Bugs: KAFKA-1178
https://issues.apache.org/jira/browse/KAFKA-1178


Repository: kafka


Description
---

1. Fixed broker to only follow become-follower/leader request if the broker 
exists in the assigned replica list 2. Changed the under replicated check to 
compare in sync replica set size with assigned replica set size instead of 
passed in replication factor 3. Fixed the fetcher id computation to not create 
more fetcher threads than necessary due to the negative (overflown) hashcode 
value. 4. Fixed some more state change logging 5. Tested partition reassignment 
several times on a cluster with several 100s of topics, verified reassignment 
completion followed by no under replicated partitions (through jmx and through 
zookeeper)


Diffs
-

  core/src/main/scala/kafka/admin/ReassignPartitionsCommand.scala 
65c04edc11a1eb2771bfd741d59886b5fef9ccbc 
  core/src/main/scala/kafka/cluster/Partition.scala 
13f48baa1bc44533c33153fde6f1c03a067d83bc 
  core/src/main/scala/kafka/server/AbstractFetcherManager.scala 
394e981c858fa53a28d3c7fcef0afbd4faf07858 
  core/src/main/scala/kafka/server/ReplicaManager.scala 
b0a39622ab91e209f9657231917a5ae30fd835f0 

Diff: https://reviews.apache.org/r/16175/diff/


Testing
---


Thanks,

Neha Narkhede



[jira] [Updated] (KAFKA-1178) Replica fetcher thread dies while becoming follower

2013-12-10 Thread Neha Narkhede (JIRA)

 [ 
https://issues.apache.org/jira/browse/KAFKA-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Neha Narkhede updated KAFKA-1178:
-

Attachment: KAFKA-1178.patch

> Replica fetcher thread dies while becoming follower
> ---
>
> Key: KAFKA-1178
> URL: https://issues.apache.org/jira/browse/KAFKA-1178
> Project: Kafka
>  Issue Type: Bug
>  Components: replication
>Affects Versions: 0.8.1
>Reporter: Neha Narkhede
>Assignee: Neha Narkhede
>Priority: Critical
> Attachments: KAFKA-1178.patch
>
>
> Replica fetcher thread dies due to 
> 2013/12/10 17:48:24.845 [ReplicaFetcherThread] [ReplicaFetcherThread--3-500], 
> Error due to 
> kafka.common.KafkaException: error processing data for partition 
> [pts-test_foo,1] offset 340
> at 
> kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$1$$anonfun$apply$mcV$sp$2.apply(AbstractFetcherThread.scala:139)
> at 
> kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$1$$anonfun$apply$mcV$sp$2.apply(AbstractFetcherThread.scala:111)
> at 
> scala.collection.immutable.HashMap$HashMap1.foreach(HashMap.scala:125)
> at 
> scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:344)
> at 
> scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:344)
> at 
> kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$1.apply$mcV$sp(AbstractFetcherThread.scala:111)
> at 
> kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$1.apply(AbstractFetcherThread.scala:111)
> at 
> kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$1.apply(AbstractFetcherThread.scala:111)
> at kafka.utils.Utils$.inLock(Utils.scala:538)
> at 
> kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:110)
> at 
> kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:88)
> at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51)
> Caused by: java.util.NoSuchElementException: None.get
> at scala.None$.get(Option.scala:185)
> at scala.None$.get(Option.scala:183)
> at 
> kafka.server.ReplicaFetcherThread.processPartitionData(ReplicaFetcherThread.scala:45)
> at 
> kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$1$$anonfun$apply$mcV$sp$2.apply(AbstractFetcherThread.scala:130)
> ... 11 more



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)


[jira] [Commented] (KAFKA-1178) Replica fetcher thread dies while becoming follower

2013-12-10 Thread Neha Narkhede (JIRA)

[ 
https://issues.apache.org/jira/browse/KAFKA-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13845126#comment-13845126
 ] 

Neha Narkhede commented on KAFKA-1178:
--

Created reviewboard https://reviews.apache.org/r/16175/
 against branch trunk

> Replica fetcher thread dies while becoming follower
> ---
>
> Key: KAFKA-1178
> URL: https://issues.apache.org/jira/browse/KAFKA-1178
> Project: Kafka
>  Issue Type: Bug
>  Components: replication
>Affects Versions: 0.8.1
>Reporter: Neha Narkhede
>Assignee: Neha Narkhede
>Priority: Critical
> Attachments: KAFKA-1178.patch
>
>
> Replica fetcher thread dies due to 
> 2013/12/10 17:48:24.845 [ReplicaFetcherThread] [ReplicaFetcherThread--3-500], 
> Error due to 
> kafka.common.KafkaException: error processing data for partition 
> [pts-test_foo,1] offset 340
> at 
> kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$1$$anonfun$apply$mcV$sp$2.apply(AbstractFetcherThread.scala:139)
> at 
> kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$1$$anonfun$apply$mcV$sp$2.apply(AbstractFetcherThread.scala:111)
> at 
> scala.collection.immutable.HashMap$HashMap1.foreach(HashMap.scala:125)
> at 
> scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:344)
> at 
> scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:344)
> at 
> kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$1.apply$mcV$sp(AbstractFetcherThread.scala:111)
> at 
> kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$1.apply(AbstractFetcherThread.scala:111)
> at 
> kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$1.apply(AbstractFetcherThread.scala:111)
> at kafka.utils.Utils$.inLock(Utils.scala:538)
> at 
> kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:110)
> at 
> kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:88)
> at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51)
> Caused by: java.util.NoSuchElementException: None.get
> at scala.None$.get(Option.scala:185)
> at scala.None$.get(Option.scala:183)
> at 
> kafka.server.ReplicaFetcherThread.processPartitionData(ReplicaFetcherThread.scala:45)
> at 
> kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$1$$anonfun$apply$mcV$sp$2.apply(AbstractFetcherThread.scala:130)
> ... 11 more



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)