[jira] [Commented] (KAFKA-1171) Gradle build for Kafka

[ https://issues.apache.org/jira/browse/KAFKA-1171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13890404#comment-13890404 ] Jun Rao commented on KAFKA-1171: Jay, Is there any benefit of not defaulting to 2.8.0? I

[jira] [Commented] (KAFKA-1171) Gradle build for Kafka

[ https://issues.apache.org/jira/browse/KAFKA-1171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13890403#comment-13890403 ] Jun Rao commented on KAFKA-1171: David, I don't think there is any outstanding issue. Cou

[jira] [Updated] (KAFKA-1164) kafka should depend on snappy 1.0.5 (instead of 1.0.4.1)

[ https://issues.apache.org/jira/browse/KAFKA-1164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede updated KAFKA-1164: - Assignee: Jason Rosenberg > kafka should depend on snappy 1.0.5 (instead of 1.0.4.1) > --

[jira] [Closed] (KAFKA-1164) kafka should depend on snappy 1.0.5 (instead of 1.0.4.1)

[ https://issues.apache.org/jira/browse/KAFKA-1164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede closed KAFKA-1164. > kafka should depend on snappy 1.0.5 (instead of 1.0.4.1) > --

[jira] [Commented] (KAFKA-1164) kafka should depend on snappy 1.0.5 (instead of 1.0.4.1)

[ https://issues.apache.org/jira/browse/KAFKA-1164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13890391#comment-13890391 ] Neha Narkhede commented on KAFKA-1164: -- +1. Thanks for the patch > kafka should depe

[jira] [Updated] (KAFKA-1164) kafka should depend on snappy 1.0.5 (instead of 1.0.4.1)

[ https://issues.apache.org/jira/browse/KAFKA-1164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede updated KAFKA-1164: - Resolution: Fixed Status: Resolved (was: Patch Available) Committed to trunk > kafka sh

Re: Review Request 17688: API refactoring for new producer

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/17688/#review33556 --- Ship it! clients/src/main/java/kafka/clients/producer/Callback.jav

[jira] [Commented] (KAFKA-1171) Gradle build for Kafka

[ https://issues.apache.org/jira/browse/KAFKA-1171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13890278#comment-13890278 ] David Arthur commented on KAFKA-1171: - I've seen some transient test failures as well,

[jira] [Commented] (KAFKA-1171) Gradle build for Kafka

[ https://issues.apache.org/jira/browse/KAFKA-1171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13890276#comment-13890276 ] Guozhang Wang commented on KAFKA-1171: -- Tried v9 with all the functionalities, a few

[jira] [Commented] (KAFKA-1238) New producer hangs in a loop detecting metadata for auto created topics

[ https://issues.apache.org/jira/browse/KAFKA-1238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13890274#comment-13890274 ] Jay Kreps commented on KAFKA-1238: -- Ack, this is clearly the wrong fix, I was misinterpre

[jira] [Commented] (KAFKA-1227) Code dump of new producer

[ https://issues.apache.org/jira/browse/KAFKA-1227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13890264#comment-13890264 ] Jay Kreps commented on KAFKA-1227: -- Created reviewboard https://reviews.apache.org/r/1768

[jira] [Updated] (KAFKA-1227) Code dump of new producer

[ https://issues.apache.org/jira/browse/KAFKA-1227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Kreps updated KAFKA-1227: - Attachment: KAFKA-1227.patch > Code dump of new producer > - > >

Review Request 17688: API refactoring for new producer

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/17688/ --- Review request for kafka. Bugs: KAFKA-1227 https://issues.apache.org/jira/b

Re: Review Request 17649: Patch for KAFKA-1237

> On Feb. 3, 2014, 9:48 p.m., Guozhang Wang wrote: > > Conditional on the producer clean shutdown :) > > Guozhang Wang wrote: > I'm wondering if you are still able to build/release with the changes in > Build.scala? In my case it will cause ambiguous reference error. > > Neha Narkhede wrot

[jira] [Commented] (KAFKA-1237) Add mirror maker using 08 consumer and 09 producer

[ https://issues.apache.org/jira/browse/KAFKA-1237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13890252#comment-13890252 ] Neha Narkhede commented on KAFKA-1237: -- Updated reviewboard https://reviews.apache.or

[jira] [Updated] (KAFKA-1237) Add mirror maker using 08 consumer and 09 producer

[ https://issues.apache.org/jira/browse/KAFKA-1237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede updated KAFKA-1237: - Attachment: KAFKA-1237_2014-02-03_17:26:37.patch > Add mirror maker using 08 consumer and 09 prod

[jira] [Commented] (KAFKA-1237) Add mirror maker using 08 consumer and 09 producer

[ https://issues.apache.org/jira/browse/KAFKA-1237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13890249#comment-13890249 ] Neha Narkhede commented on KAFKA-1237: -- Updated reviewboard https://reviews.apache.or

Re: Review Request 17649: Patch for KAFKA-1237

> On Feb. 3, 2014, 9:48 p.m., Guozhang Wang wrote: > > Conditional on the producer clean shutdown :) > > Guozhang Wang wrote: > I'm wondering if you are still able to build/release with the changes in > Build.scala? In my case it will cause ambiguous reference error. yes, I'm able to build

[jira] [Updated] (KAFKA-1237) Add mirror maker using 08 consumer and 09 producer

[ https://issues.apache.org/jira/browse/KAFKA-1237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede updated KAFKA-1237: - Attachment: KAFKA-1237_2014-02-03_17:25:11.patch > Add mirror maker using 08 consumer and 09 prod

Re: Review Request 17649: Patch for KAFKA-1237

> On Feb. 3, 2014, 7:55 p.m., Joel Koshy wrote: > > core/src/main/scala/kafka/tools/newproducer/MirrorMaker.java, line 81 > > > > > > Can you fix the whitespace issues? I'm actually not sure what's introducing the whit

Re: Review Request 17649: Patch for KAFKA-1237

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/17649/ --- (Updated Feb. 4, 2014, 1:26 a.m.) Review request for kafka. Bugs: KAFKA-1237

Re: Review Request 17649: Patch for KAFKA-1237

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/17649/ --- (Updated Feb. 4, 2014, 1:25 a.m.) Review request for kafka. Bugs: KAFKA-1237

[jira] [Updated] (KAFKA-1237) Add mirror maker using 08 consumer and 09 producer

[ https://issues.apache.org/jira/browse/KAFKA-1237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede updated KAFKA-1237: - Attachment: KAFKA-1237_2014-02-03_17:22:08.patch > Add mirror maker using 08 consumer and 09 prod

[jira] [Commented] (KAFKA-1237) Add mirror maker using 08 consumer and 09 producer

[ https://issues.apache.org/jira/browse/KAFKA-1237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13890246#comment-13890246 ] Neha Narkhede commented on KAFKA-1237: -- Updated reviewboard https://reviews.apache.or

Re: Review Request 17649: Patch for KAFKA-1237

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/17649/ --- (Updated Feb. 4, 2014, 1:22 a.m.) Review request for kafka. Bugs: KAFKA-1237

Re: Review Request 17649: Patch for KAFKA-1237

> On Feb. 3, 2014, 9:48 p.m., Guozhang Wang wrote: > > Conditional on the producer clean shutdown :) I'm wondering if you are still able to build/release with the changes in Build.scala? In my case it will cause ambiguous reference error. - Guozhang -

[jira] [Commented] (KAFKA-1174) Empty jar in Maven Central for Scala 2.8.0

[ https://issues.apache.org/jira/browse/KAFKA-1174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13890061#comment-13890061 ] Brent Bradbury commented on KAFKA-1174: --- +1, burned ourselves on JRuby with this for

Re: Logging in new clients

Alright, I'm sold. If no objections, slf4j it is. -Jay On Mon, Feb 3, 2014 at 1:11 PM, Joel Koshy wrote: > > I see server and client logging and dependencies are really different in > my > > mind. The server is a standalone process. Once we decouple the clients I > > think we can actually be a

Re: Proposed Changes To New Producer Public API

For (3) we could also do the following: - On any retryable producer error, force a metadata refresh (in handleProducerResponse). - In handleMetadataResponse, the producer can (internally) close out connections that are no longer valid. (i.e., connections to {old set of leader brokers} - {new

Re: Review Request 17653: Refactor API of new producer

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/17653/#review33514 --- clients/src/main/java/kafka/clients/producer/RecordMetadata.java

Re: Review Request 17524: Add one integration test for the new producer

> On Jan. 30, 2014, 5:02 p.m., Jay Kreps wrote: > > This is a very nice test! It would be good to also have a scenario that has > > two servers with a replication-factor two topic and kills one of the > > servers to validate the reconnect/metadata logic. Yeah I plan to have the replication sce

Re: Review Request 17524: Add one integration test for the new producer

> On Jan. 30, 2014, 6:12 p.m., Jun Rao wrote: > > Could you build a release tar and make sure that we don't drag in the new > > client jar as a dependency? ./sbt release-tar failed due to ambiguous references: [error] it is imported twice in the same scope by [error] import kafka.common._ [err

Re: Review Request 17524: Add one integration test for the new producer

> On Jan. 30, 2014, 6:12 p.m., Jun Rao wrote: > > core/src/test/scala/integration/kafka/api/ProducerSendTest.scala, lines > > 75-76 > > > > > > Do we really want to print those out in an integration test? I used this

Re: Review Request 17649: Patch for KAFKA-1237

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/17649/#review33511 --- Ship it! Conditional on the producer clean shutdown :) - Guozhang

[jira] [Commented] (KAFKA-1032) Messages sent to the old leader will be lost on broker GC resulted failure

[ https://issues.apache.org/jira/browse/KAFKA-1032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13889936#comment-13889936 ] Guozhang Wang commented on KAFKA-1032: -- I think this can wait until 0.9 > Messages s

Re: Logging in new clients

> I see server and client logging and dependencies are really different in my > mind. The server is a standalone process. Once we decouple the clients I > think we can actually be a lot more aggressive about using more libraries > on the server, why not? We have avoided that so far because they ine

Re: Logging in new clients

+1 for SLF4J. It is the only option that doesn't force a consumer to use a particular style of configuration. It is common enough that I think it would not cause any adoption friction. There's a relevant and lengthier discussion here: http://stackoverflow.com/questions/11359187/dont-get-it-why-not-

[jira] [Commented] (KAFKA-1125) Add options to let admin tools block until finish

[ https://issues.apache.org/jira/browse/KAFKA-1125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13889898#comment-13889898 ] Todd Palino commented on KAFKA-1125: I believe an option for each of these tools to ma

[jira] [Commented] (KAFKA-1171) Gradle build for Kafka

[ https://issues.apache.org/jira/browse/KAFKA-1171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13889860#comment-13889860 ] Jun Rao commented on KAFKA-1171: Please use "git apply kafka-1171_v9.patch" to try the pat

[jira] [Updated] (KAFKA-1171) Gradle build for Kafka

[ https://issues.apache.org/jira/browse/KAFKA-1171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao updated KAFKA-1171: --- Attachment: kafka-1171_v9.patch Attach patch v9. The pom dependency seems to work since the included test de

Re: Review Request 17671: Enable kafka server to keep retrying controlled shutdown with value -1

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/17671/ --- (Updated Feb. 3, 2014, 8:07 p.m.) Review request for kafka. Bugs: KAFKA-1235

Re: Review Request 17649: Patch for KAFKA-1237

> On Feb. 3, 2014, 7:55 p.m., Joel Koshy wrote: > > core/src/main/scala/kafka/tools/newproducer/MirrorMaker.java, line 118 > > > > > > Can we try just one producer thread? Also, if you do go with multiple > > producer

[jira] [Updated] (KAFKA-1235) Enable server to indefinitely retry on controlled shutdown

[ https://issues.apache.org/jira/browse/KAFKA-1235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-1235: - Attachment: KAFKA-1235.patch > Enable server to indefinitely retry on controlled shutdown > -

[jira] [Commented] (KAFKA-1235) Enable server to indefinitely retry on controlled shutdown

[ https://issues.apache.org/jira/browse/KAFKA-1235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13889844#comment-13889844 ] Guozhang Wang commented on KAFKA-1235: -- Created reviewboard https://reviews.apache.or

Review Request 17671: Enable kafka server to keep retrying controlled shutdown with value -1

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/17671/ --- Review request for kafka. Bugs: KAFKA-1235 https://issues.apache.org/jira/b

Re: Logging in new clients

Hey Joel, I see server and client logging and dependencies are really different in my mind. The server is a standalone process. Once we decouple the clients I think we can actually be a lot more aggressive about using more libraries on the server, why not? We have avoided that so far because they

Re: Review Request 17479: Patch for KAFKA-1232

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/17479/#review33484 --- core/src/main/scala/kafka/admin/TopicCommand.scala

Re: Review Request 17649: Patch for KAFKA-1237

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/17649/#review33480 --- core/src/main/scala/kafka/tools/newproducer/MirrorMaker.java

Re: Proposed Changes To New Producer Public API

1. Yes, I think the name could be improved. However that name doesn't really express what it does. What about RecordSentCallback? 2. As Neha said the nodes are initialized to a random position. Round robin is preferable to random (lower variance, cheaper to compute, etc). Your point about skipping

Re: Logging in new clients

My vote would be with log4j, I don't have that much experience with log4j2 or a good feel for how much the industry is moving towards it. On Mon, Feb 3, 2014 at 11:17 AM, Joel Koshy wrote: > We are already using other libraries in various parts of our code > (e.g., metrics, zkclient, joptsimpl

Re: Logging in new clients

I'm all for less dependencies but I would personally vote for slf4j-api. Just don't use any underlying implementations like logback or slf4j-log4j bridge. Then people can hookup whatever they want. On Mon, Feb 3, 2014 at 11:08 AM, Neha Narkhede wrote: > >> Basically my preference would be java.

Re: Proposed Changes To New Producer Public API

>> (1) if the replication factor is 1 and there is a broker failure, we can still route the message, (2) if a bunch of producers are started at the same time, this prevents them from picking up the same partition in a synchronized way. You raised a good point, Jun. Regarding #1, shouldn't round ro

Re: Logging in new clients

We are already using other libraries in various parts of our code (e.g., metrics, zkclient, joptsimple, etc) some of which pull in these other logging dependencies anyway. i.e., what do we gain by using jul? There may be a good reason why people don't use jul so I think we should fully understand t

Re: Logging in new clients

>> Basically my preference would be java.util.logging unless there is some known problem with it, otherwise I guess slf4j, and if not that then log4j. +1. My preference is to use java.util.logging to avoid adding an external dependency, but I'm not too sure about what's the "standard" out there, s

[jira] [Commented] (KAFKA-1238) New producer hangs in a loop detecting metadata for auto created topics

[ https://issues.apache.org/jira/browse/KAFKA-1238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13889759#comment-13889759 ] Neha Narkhede commented on KAFKA-1238: -- Thanks for the patch, Jay! Review comments-

Logging in new clients

We probably need to add a small amount of logging in the new producer and (soon) consumer clients. I wanted to have a quick discussion on logging libraries before I start adding this in the producer. Previously we have been pretty verbose loggers and I think we should stop that. For clients you mo

[jira] Subscription: outstanding kafka patches

Issue Subscription Filter: outstanding kafka patches (86 issues) The list of outstanding kafka patches Subscriber: kafka-mailing-list Key Summary KAFKA-1238 New producer hangs in a loop detecting metadata for auto created topics https://issues.apache.org/jira/browse/KAFKA-123

Re: Proposed Changes To New Producer Public API

Fine with most of the changes. Just a few questions/comments. 1. Would it be better to change Callback to ProducerCallback to distinguish it from controller callback and potential future other types of callbacks (e.g. consumer)? 2. If no key is present AND no partition id is present, partitions w

Re: Proposed Changes To New Producer Public API

Also, I thought a bit more about the org.apache thing. This is clearly an aesthetic thing, but I do feel aesthetics are important. The arguments for the kafka.* package is basically that putting your domain name in the namespace is a silly cargo cult thing that Java people do. I have never seen a

[jira] [Commented] (KAFKA-1171) Gradle build for Kafka

[ https://issues.apache.org/jira/browse/KAFKA-1171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13889719#comment-13889719 ] Guozhang Wang commented on KAFKA-1171: -- No, when I applied the patch these are the fi

[jira] [Updated] (KAFKA-1238) New producer hangs in a loop detecting metadata for auto created topics

[ https://issues.apache.org/jira/browse/KAFKA-1238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Kreps updated KAFKA-1238: - Attachment: KAFKA-1238.patch Fix attached. > New producer hangs in a loop detecting metadata for auto cr

[jira] [Commented] (KAFKA-1171) Gradle build for Kafka

[ https://issues.apache.org/jira/browse/KAFKA-1171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13889711#comment-13889711 ] Jun Rao commented on KAFKA-1171: Guozhang, That class is included in the patch in gradle/

[jira] [Commented] (KAFKA-1171) Gradle build for Kafka

[ https://issues.apache.org/jira/browse/KAFKA-1171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13889705#comment-13889705 ] Guozhang Wang commented on KAFKA-1171: -- Used Linux to try the patch, still get the fo

[jira] [Commented] (KAFKA-1171) Gradle build for Kafka

[ https://issues.apache.org/jira/browse/KAFKA-1171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13889696#comment-13889696 ] Jay Kreps commented on KAFKA-1171: -- Shouldn't we default to scala 2.9 or something newer

[jira] [Updated] (KAFKA-1171) Gradle build for Kafka

[ https://issues.apache.org/jira/browse/KAFKA-1171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao updated KAFKA-1171: --- Attachment: kafka-1171_v8.patch Attach patch v8. 1. ./gradlew idea works now. I still couldn't get it compil

Re: code layout

That's why I was thinking c) makes sense from a design perspective. > On Feb 2, 2014, at 21:59, Jay Kreps wrote: > > Yes, this is a good point, the admin api could depend on zookeeper. It does > today. That would not work in the existing clients package. > > -Jay > > >> On Sun, Feb 2, 2014 at

[jira] [Commented] (KAFKA-334) Some tests fail when building on a Windows box

[ https://issues.apache.org/jira/browse/KAFKA-334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13889563#comment-13889563 ] Roman Garcia commented on KAFKA-334: Now the flow I described works, but doesn't look l

Re: Review Request 17653: Refactor API of new producer

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/17653/#review33423 --- clients/src/main/java/kafka/common/Cluster.java

Re: Review Request 17653: Refactor API of new producer

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/17653/#review33422 --- clients/src/main/java/kafka/common/Cluster.java