Re: Review Request 17653: Refactor API of new producer

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/17653/#review33422 --- clients/src/main/java/kafka/common/Cluster.java

Re: Review Request 17653: Refactor API of new producer

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/17653/#review33423 --- clients/src/main/java/kafka/common/Cluster.java

[jira] [Commented] (KAFKA-334) Some tests fail when building on a Windows box

[ https://issues.apache.org/jira/browse/KAFKA-334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13889563#comment-13889563 ] Roman Garcia commented on KAFKA-334: Now the flow I described works, but doesn't look l

Re: code layout

That's why I was thinking c) makes sense from a design perspective. > On Feb 2, 2014, at 21:59, Jay Kreps wrote: > > Yes, this is a good point, the admin api could depend on zookeeper. It does > today. That would not work in the existing clients package. > > -Jay > > >> On Sun, Feb 2, 2014 at

[jira] [Updated] (KAFKA-1171) Gradle build for Kafka

[ https://issues.apache.org/jira/browse/KAFKA-1171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao updated KAFKA-1171: --- Attachment: kafka-1171_v8.patch Attach patch v8. 1. ./gradlew idea works now. I still couldn't get it compil

[jira] [Commented] (KAFKA-1171) Gradle build for Kafka

[ https://issues.apache.org/jira/browse/KAFKA-1171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13889696#comment-13889696 ] Jay Kreps commented on KAFKA-1171: -- Shouldn't we default to scala 2.9 or something newer

[jira] [Commented] (KAFKA-1171) Gradle build for Kafka

[ https://issues.apache.org/jira/browse/KAFKA-1171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13889705#comment-13889705 ] Guozhang Wang commented on KAFKA-1171: -- Used Linux to try the patch, still get the fo

[jira] [Commented] (KAFKA-1171) Gradle build for Kafka

[ https://issues.apache.org/jira/browse/KAFKA-1171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13889711#comment-13889711 ] Jun Rao commented on KAFKA-1171: Guozhang, That class is included in the patch in gradle/

[jira] [Commented] (KAFKA-1171) Gradle build for Kafka

[ https://issues.apache.org/jira/browse/KAFKA-1171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13889719#comment-13889719 ] Guozhang Wang commented on KAFKA-1171: -- No, when I applied the patch these are the fi

[jira] [Updated] (KAFKA-1238) New producer hangs in a loop detecting metadata for auto created topics

[ https://issues.apache.org/jira/browse/KAFKA-1238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Kreps updated KAFKA-1238: - Attachment: KAFKA-1238.patch Fix attached. > New producer hangs in a loop detecting metadata for auto cr

Re: Proposed Changes To New Producer Public API

Also, I thought a bit more about the org.apache thing. This is clearly an aesthetic thing, but I do feel aesthetics are important. The arguments for the kafka.* package is basically that putting your domain name in the namespace is a silly cargo cult thing that Java people do. I have never seen a

Re: Proposed Changes To New Producer Public API

Fine with most of the changes. Just a few questions/comments. 1. Would it be better to change Callback to ProducerCallback to distinguish it from controller callback and potential future other types of callbacks (e.g. consumer)? 2. If no key is present AND no partition id is present, partitions w

[jira] Subscription: outstanding kafka patches

Issue Subscription Filter: outstanding kafka patches (86 issues) The list of outstanding kafka patches Subscriber: kafka-mailing-list Key Summary KAFKA-1238 New producer hangs in a loop detecting metadata for auto created topics https://issues.apache.org/jira/browse/KAFKA-123

Logging in new clients

We probably need to add a small amount of logging in the new producer and (soon) consumer clients. I wanted to have a quick discussion on logging libraries before I start adding this in the producer. Previously we have been pretty verbose loggers and I think we should stop that. For clients you mo

[jira] [Commented] (KAFKA-1238) New producer hangs in a loop detecting metadata for auto created topics

[ https://issues.apache.org/jira/browse/KAFKA-1238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13889759#comment-13889759 ] Neha Narkhede commented on KAFKA-1238: -- Thanks for the patch, Jay! Review comments-

Re: Logging in new clients

>> Basically my preference would be java.util.logging unless there is some known problem with it, otherwise I guess slf4j, and if not that then log4j. +1. My preference is to use java.util.logging to avoid adding an external dependency, but I'm not too sure about what's the "standard" out there, s

Re: Logging in new clients

We are already using other libraries in various parts of our code (e.g., metrics, zkclient, joptsimple, etc) some of which pull in these other logging dependencies anyway. i.e., what do we gain by using jul? There may be a good reason why people don't use jul so I think we should fully understand t

Re: Proposed Changes To New Producer Public API

>> (1) if the replication factor is 1 and there is a broker failure, we can still route the message, (2) if a bunch of producers are started at the same time, this prevents them from picking up the same partition in a synchronized way. You raised a good point, Jun. Regarding #1, shouldn't round ro

Re: Logging in new clients

I'm all for less dependencies but I would personally vote for slf4j-api. Just don't use any underlying implementations like logback or slf4j-log4j bridge. Then people can hookup whatever they want. On Mon, Feb 3, 2014 at 11:08 AM, Neha Narkhede wrote: > >> Basically my preference would be java.

Re: Logging in new clients

My vote would be with log4j, I don't have that much experience with log4j2 or a good feel for how much the industry is moving towards it. On Mon, Feb 3, 2014 at 11:17 AM, Joel Koshy wrote: > We are already using other libraries in various parts of our code > (e.g., metrics, zkclient, joptsimpl

Re: Proposed Changes To New Producer Public API

1. Yes, I think the name could be improved. However that name doesn't really express what it does. What about RecordSentCallback? 2. As Neha said the nodes are initialized to a random position. Round robin is preferable to random (lower variance, cheaper to compute, etc). Your point about skipping

Re: Review Request 17649: Patch for KAFKA-1237

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/17649/#review33480 --- core/src/main/scala/kafka/tools/newproducer/MirrorMaker.java

Re: Review Request 17479: Patch for KAFKA-1232

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/17479/#review33484 --- core/src/main/scala/kafka/admin/TopicCommand.scala

Re: Logging in new clients

Hey Joel, I see server and client logging and dependencies are really different in my mind. The server is a standalone process. Once we decouple the clients I think we can actually be a lot more aggressive about using more libraries on the server, why not? We have avoided that so far because they

Review Request 17671: Enable kafka server to keep retrying controlled shutdown with value -1

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/17671/ --- Review request for kafka. Bugs: KAFKA-1235 https://issues.apache.org/jira/b

[jira] [Commented] (KAFKA-1235) Enable server to indefinitely retry on controlled shutdown

[ https://issues.apache.org/jira/browse/KAFKA-1235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13889844#comment-13889844 ] Guozhang Wang commented on KAFKA-1235: -- Created reviewboard https://reviews.apache.or

[jira] [Updated] (KAFKA-1235) Enable server to indefinitely retry on controlled shutdown

[ https://issues.apache.org/jira/browse/KAFKA-1235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-1235: - Attachment: KAFKA-1235.patch > Enable server to indefinitely retry on controlled shutdown > -

Re: Review Request 17649: Patch for KAFKA-1237

> On Feb. 3, 2014, 7:55 p.m., Joel Koshy wrote: > > core/src/main/scala/kafka/tools/newproducer/MirrorMaker.java, line 118 > > > > > > Can we try just one producer thread? Also, if you do go with multiple > > producer

Re: Review Request 17671: Enable kafka server to keep retrying controlled shutdown with value -1

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/17671/ --- (Updated Feb. 3, 2014, 8:07 p.m.) Review request for kafka. Bugs: KAFKA-1235

[jira] [Updated] (KAFKA-1171) Gradle build for Kafka

[ https://issues.apache.org/jira/browse/KAFKA-1171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao updated KAFKA-1171: --- Attachment: kafka-1171_v9.patch Attach patch v9. The pom dependency seems to work since the included test de

[jira] [Commented] (KAFKA-1171) Gradle build for Kafka

[ https://issues.apache.org/jira/browse/KAFKA-1171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13889860#comment-13889860 ] Jun Rao commented on KAFKA-1171: Please use "git apply kafka-1171_v9.patch" to try the pat

[jira] [Commented] (KAFKA-1125) Add options to let admin tools block until finish

[ https://issues.apache.org/jira/browse/KAFKA-1125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13889898#comment-13889898 ] Todd Palino commented on KAFKA-1125: I believe an option for each of these tools to ma

Re: Logging in new clients

+1 for SLF4J. It is the only option that doesn't force a consumer to use a particular style of configuration. It is common enough that I think it would not cause any adoption friction. There's a relevant and lengthier discussion here: http://stackoverflow.com/questions/11359187/dont-get-it-why-not-

Re: Logging in new clients

> I see server and client logging and dependencies are really different in my > mind. The server is a standalone process. Once we decouple the clients I > think we can actually be a lot more aggressive about using more libraries > on the server, why not? We have avoided that so far because they ine

[jira] [Commented] (KAFKA-1032) Messages sent to the old leader will be lost on broker GC resulted failure

[ https://issues.apache.org/jira/browse/KAFKA-1032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13889936#comment-13889936 ] Guozhang Wang commented on KAFKA-1032: -- I think this can wait until 0.9 > Messages s

Re: Review Request 17649: Patch for KAFKA-1237

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/17649/#review33511 --- Ship it! Conditional on the producer clean shutdown :) - Guozhang

Re: Review Request 17524: Add one integration test for the new producer

> On Jan. 30, 2014, 6:12 p.m., Jun Rao wrote: > > core/src/test/scala/integration/kafka/api/ProducerSendTest.scala, lines > > 75-76 > > > > > > Do we really want to print those out in an integration test? I used this

Re: Review Request 17524: Add one integration test for the new producer

> On Jan. 30, 2014, 5:02 p.m., Jay Kreps wrote: > > This is a very nice test! It would be good to also have a scenario that has > > two servers with a replication-factor two topic and kills one of the > > servers to validate the reconnect/metadata logic. Yeah I plan to have the replication sce

Re: Review Request 17524: Add one integration test for the new producer

> On Jan. 30, 2014, 6:12 p.m., Jun Rao wrote: > > Could you build a release tar and make sure that we don't drag in the new > > client jar as a dependency? ./sbt release-tar failed due to ambiguous references: [error] it is imported twice in the same scope by [error] import kafka.common._ [err

Re: Review Request 17653: Refactor API of new producer

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/17653/#review33514 --- clients/src/main/java/kafka/clients/producer/RecordMetadata.java

Re: Proposed Changes To New Producer Public API

For (3) we could also do the following: - On any retryable producer error, force a metadata refresh (in handleProducerResponse). - In handleMetadataResponse, the producer can (internally) close out connections that are no longer valid. (i.e., connections to {old set of leader brokers} - {new

Re: Logging in new clients

Alright, I'm sold. If no objections, slf4j it is. -Jay On Mon, Feb 3, 2014 at 1:11 PM, Joel Koshy wrote: > > I see server and client logging and dependencies are really different in > my > > mind. The server is a standalone process. Once we decouple the clients I > > think we can actually be a

[jira] [Commented] (KAFKA-1174) Empty jar in Maven Central for Scala 2.8.0

[ https://issues.apache.org/jira/browse/KAFKA-1174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13890061#comment-13890061 ] Brent Bradbury commented on KAFKA-1174: --- +1, burned ourselves on JRuby with this for

Re: Review Request 17649: Patch for KAFKA-1237

> On Feb. 3, 2014, 9:48 p.m., Guozhang Wang wrote: > > Conditional on the producer clean shutdown :) I'm wondering if you are still able to build/release with the changes in Build.scala? In my case it will cause ambiguous reference error. - Guozhang -

Re: Review Request 17649: Patch for KAFKA-1237

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/17649/ --- (Updated Feb. 4, 2014, 1:22 a.m.) Review request for kafka. Bugs: KAFKA-1237

[jira] [Updated] (KAFKA-1237) Add mirror maker using 08 consumer and 09 producer

[ https://issues.apache.org/jira/browse/KAFKA-1237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede updated KAFKA-1237: - Attachment: KAFKA-1237_2014-02-03_17:22:08.patch > Add mirror maker using 08 consumer and 09 prod

[jira] [Commented] (KAFKA-1237) Add mirror maker using 08 consumer and 09 producer

[ https://issues.apache.org/jira/browse/KAFKA-1237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13890246#comment-13890246 ] Neha Narkhede commented on KAFKA-1237: -- Updated reviewboard https://reviews.apache.or

Re: Review Request 17649: Patch for KAFKA-1237

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/17649/ --- (Updated Feb. 4, 2014, 1:25 a.m.) Review request for kafka. Bugs: KAFKA-1237

Re: Review Request 17649: Patch for KAFKA-1237

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/17649/ --- (Updated Feb. 4, 2014, 1:26 a.m.) Review request for kafka. Bugs: KAFKA-1237

[jira] [Commented] (KAFKA-1237) Add mirror maker using 08 consumer and 09 producer

[ https://issues.apache.org/jira/browse/KAFKA-1237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13890252#comment-13890252 ] Neha Narkhede commented on KAFKA-1237: -- Updated reviewboard https://reviews.apache.or

[jira] [Updated] (KAFKA-1237) Add mirror maker using 08 consumer and 09 producer

[ https://issues.apache.org/jira/browse/KAFKA-1237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede updated KAFKA-1237: - Attachment: KAFKA-1237_2014-02-03_17:26:37.patch > Add mirror maker using 08 consumer and 09 prod

[jira] [Commented] (KAFKA-1237) Add mirror maker using 08 consumer and 09 producer

[ https://issues.apache.org/jira/browse/KAFKA-1237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13890249#comment-13890249 ] Neha Narkhede commented on KAFKA-1237: -- Updated reviewboard https://reviews.apache.or

Re: Review Request 17649: Patch for KAFKA-1237

> On Feb. 3, 2014, 9:48 p.m., Guozhang Wang wrote: > > Conditional on the producer clean shutdown :) > > Guozhang Wang wrote: > I'm wondering if you are still able to build/release with the changes in > Build.scala? In my case it will cause ambiguous reference error. yes, I'm able to build

[jira] [Updated] (KAFKA-1237) Add mirror maker using 08 consumer and 09 producer

[ https://issues.apache.org/jira/browse/KAFKA-1237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede updated KAFKA-1237: - Attachment: KAFKA-1237_2014-02-03_17:25:11.patch > Add mirror maker using 08 consumer and 09 prod

Re: Review Request 17649: Patch for KAFKA-1237

> On Feb. 3, 2014, 7:55 p.m., Joel Koshy wrote: > > core/src/main/scala/kafka/tools/newproducer/MirrorMaker.java, line 81 > > > > > > Can you fix the whitespace issues? I'm actually not sure what's introducing the whit

Re: Review Request 17649: Patch for KAFKA-1237

> On Feb. 3, 2014, 9:48 p.m., Guozhang Wang wrote: > > Conditional on the producer clean shutdown :) > > Guozhang Wang wrote: > I'm wondering if you are still able to build/release with the changes in > Build.scala? In my case it will cause ambiguous reference error. > > Neha Narkhede wrot

Review Request 17688: API refactoring for new producer

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/17688/ --- Review request for kafka. Bugs: KAFKA-1227 https://issues.apache.org/jira/b

[jira] [Updated] (KAFKA-1227) Code dump of new producer

[ https://issues.apache.org/jira/browse/KAFKA-1227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Kreps updated KAFKA-1227: - Attachment: KAFKA-1227.patch > Code dump of new producer > - > >

[jira] [Commented] (KAFKA-1227) Code dump of new producer

[ https://issues.apache.org/jira/browse/KAFKA-1227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13890264#comment-13890264 ] Jay Kreps commented on KAFKA-1227: -- Created reviewboard https://reviews.apache.org/r/1768

[jira] [Commented] (KAFKA-1238) New producer hangs in a loop detecting metadata for auto created topics

[ https://issues.apache.org/jira/browse/KAFKA-1238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13890274#comment-13890274 ] Jay Kreps commented on KAFKA-1238: -- Ack, this is clearly the wrong fix, I was misinterpre

[jira] [Commented] (KAFKA-1171) Gradle build for Kafka

[ https://issues.apache.org/jira/browse/KAFKA-1171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13890276#comment-13890276 ] Guozhang Wang commented on KAFKA-1171: -- Tried v9 with all the functionalities, a few

[jira] [Commented] (KAFKA-1171) Gradle build for Kafka

[ https://issues.apache.org/jira/browse/KAFKA-1171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13890278#comment-13890278 ] David Arthur commented on KAFKA-1171: - I've seen some transient test failures as well,

Re: Review Request 17688: API refactoring for new producer

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/17688/#review33556 --- Ship it! clients/src/main/java/kafka/clients/producer/Callback.jav

[jira] [Closed] (KAFKA-1164) kafka should depend on snappy 1.0.5 (instead of 1.0.4.1)

[ https://issues.apache.org/jira/browse/KAFKA-1164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede closed KAFKA-1164. > kafka should depend on snappy 1.0.5 (instead of 1.0.4.1) > --

[jira] [Commented] (KAFKA-1164) kafka should depend on snappy 1.0.5 (instead of 1.0.4.1)

[ https://issues.apache.org/jira/browse/KAFKA-1164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13890391#comment-13890391 ] Neha Narkhede commented on KAFKA-1164: -- +1. Thanks for the patch > kafka should depe

[jira] [Updated] (KAFKA-1164) kafka should depend on snappy 1.0.5 (instead of 1.0.4.1)

[ https://issues.apache.org/jira/browse/KAFKA-1164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede updated KAFKA-1164: - Resolution: Fixed Status: Resolved (was: Patch Available) Committed to trunk > kafka sh

[jira] [Updated] (KAFKA-1164) kafka should depend on snappy 1.0.5 (instead of 1.0.4.1)

[ https://issues.apache.org/jira/browse/KAFKA-1164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede updated KAFKA-1164: - Assignee: Jason Rosenberg > kafka should depend on snappy 1.0.5 (instead of 1.0.4.1) > --

[jira] [Commented] (KAFKA-1171) Gradle build for Kafka

[ https://issues.apache.org/jira/browse/KAFKA-1171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13890403#comment-13890403 ] Jun Rao commented on KAFKA-1171: David, I don't think there is any outstanding issue. Cou

[jira] [Commented] (KAFKA-1171) Gradle build for Kafka

[ https://issues.apache.org/jira/browse/KAFKA-1171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13890404#comment-13890404 ] Jun Rao commented on KAFKA-1171: Jay, Is there any benefit of not defaulting to 2.8.0? I