Re: Proposed Changes To New Producer Public API

2014-02-02 Thread Jay Kreps
Okay I posted a patch against trunk that carries out the refactoring described above: https://issues.apache.org/jira/browse/KAFKA-1227 Updated javadoc is here: http://empathybox.com/kafka-javadoc This touches a fair number of files as I also improved documentation and standardized terminology in

Re: code layout

2014-02-02 Thread Jay Kreps
Yes, this is a good point, the admin api could depend on zookeeper. It does today. That would not work in the existing clients package. -Jay On Sun, Feb 2, 2014 at 4:20 PM, Steve Morin wrote: > Would the admin api's potentially have different dependencies? > > > On Feb 1, 2014, at 15:06, Neha

[jira] [Updated] (KAFKA-1227) Code dump of new producer

2014-02-02 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Kreps updated KAFKA-1227: - Attachment: KAFKA-1227.patch > Code dump of new producer > - > >

[jira] [Commented] (KAFKA-1227) Code dump of new producer

2014-02-02 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13889247#comment-13889247 ] Jay Kreps commented on KAFKA-1227: -- Created reviewboard https://reviews.apache.org/r/1765

Review Request 17653: Refactor API of new producer

2014-02-02 Thread Jay Kreps
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/17653/ --- Review request for kafka. Bugs: KAFKA-1227 https://issues.apache.org/jira/b

[jira] [Updated] (KAFKA-1171) Gradle build for Kafka

2014-02-02 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao updated KAFKA-1171: --- Attachment: kafka-1171_v7.patch Attach patch v7. Additionally fixes since v6. 1. defaulted the build to scala

[jira] [Commented] (KAFKA-1237) Add mirror maker using 08 consumer and 09 producer

2014-02-02 Thread Neha Narkhede (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13889146#comment-13889146 ] Neha Narkhede commented on KAFKA-1237: -- Created reviewboard https://reviews.apache.or

[jira] [Updated] (KAFKA-1237) Add mirror maker using 08 consumer and 09 producer

2014-02-02 Thread Neha Narkhede (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede updated KAFKA-1237: - Attachment: KAFKA-1237.patch > Add mirror maker using 08 consumer and 09 producer > -

Review Request 17649: Patch for KAFKA-1237

2014-02-02 Thread Neha Narkhede
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/17649/ --- Review request for kafka. Bugs: KAFKA-1237 https://issues.apache.org/jira/b

Re: Review Request 17479: Patch for KAFKA-1232

2014-02-02 Thread Guozhang Wang
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/17479/#review33404 --- Ship it! Ship It! - Guozhang Wang On Feb. 2, 2014, 7:24 p.m., Ju

Re: Proposed Changes To New Producer Public API

2014-02-02 Thread Guozhang Wang
About serialization, I am wondering if most people would try to have a customized partioning mainly due to logic motivations, such that they want some messages to go to the same partition; if that is true, they do not really care about the actually partition id but all they would do is specify the

[jira] [Commented] (KAFKA-1237) Add mirror maker using 08 consumer and 09 producer

2014-02-02 Thread Neha Narkhede (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13889121#comment-13889121 ] Neha Narkhede commented on KAFKA-1237: -- Created reviewboard against branch trunk >

[jira] [Updated] (KAFKA-1237) Add mirror maker using 08 consumer and 09 producer

2014-02-02 Thread Neha Narkhede (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede updated KAFKA-1237: - Attachment: KAFKA-1237.patch > Add mirror maker using 08 consumer and 09 producer > -

Review Request 17648: Patch for KAFKA-1237

2014-02-02 Thread Neha Narkhede
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/17648/ --- Review request for kafka. Bugs: KAFKA-1237 https://issues.apache.org/jira/b

Re: code layout

2014-02-02 Thread Steve Morin
Would the admin api's potentially have different dependencies? > On Feb 1, 2014, at 15:06, Neha Narkhede wrote: > > I like A. It will be simpler to maintain and evolve when we add the admin > APIs. > > > >> On Fri, Jan 31, 2014 at 7:32 PM, Steve Morin wrote: >> >> I like A or C >> >> I thi

Re: code layout

2014-02-02 Thread Guozhang Wang
I would vote for A just for simplicity of dependencies. On Sat, Feb 1, 2014 at 3:06 PM, Neha Narkhede wrote: > I like A. It will be simpler to maintain and evolve when we add the admin > APIs. > > > > On Fri, Jan 31, 2014 at 7:32 PM, Steve Morin wrote: > > > I like A or C > > > > I think it wil

[jira] [Created] (KAFKA-1238) New producer hangs in a loop detecting metadata for auto created topics

2014-02-02 Thread Neha Narkhede (JIRA)
Neha Narkhede created KAFKA-1238: Summary: New producer hangs in a loop detecting metadata for auto created topics Key: KAFKA-1238 URL: https://issues.apache.org/jira/browse/KAFKA-1238 Project: Kafka

[jira] [Assigned] (KAFKA-1237) Add mirror maker using 08 consumer and 09 producer

2014-02-02 Thread Neha Narkhede (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede reassigned KAFKA-1237: Assignee: Neha Narkhede > Add mirror maker using 08 consumer and 09 producer >

[jira] [Updated] (KAFKA-1164) kafka should depend on snappy 1.0.5 (instead of 1.0.4.1)

2014-02-02 Thread Jason Rosenberg (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Rosenberg updated KAFKA-1164: --- Attachment: kafka-1164.patch Here's the actual patch file (not sure why the jira 'submit patc

[jira] [Commented] (KAFKA-1180) WhiteList topic filter gets a NullPointerException on complex Regex

2014-02-02 Thread Jason Rosenberg (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13889041#comment-13889041 ] Jason Rosenberg commented on KAFKA-1180: Any updates on getting this merged into t

[jira] [Updated] (KAFKA-1164) kafka should depend on snappy 1.0.5 (instead of 1.0.4.1)

2014-02-02 Thread Jason Rosenberg (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Rosenberg updated KAFKA-1164: --- Status: Patch Available (was: Open) Here's a patch, which updates the pom dependency for sna

[jira] [Commented] (KAFKA-1164) kafka should depend on snappy 1.0.5 (instead of 1.0.4.1)

2014-02-02 Thread Neha Narkhede (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13889031#comment-13889031 ] Neha Narkhede commented on KAFKA-1164: -- Sure, would you like to submit a patch? > ka

[jira] [Updated] (KAFKA-1164) kafka should depend on snappy 1.0.5 (instead of 1.0.4.1)

2014-02-02 Thread Neha Narkhede (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede updated KAFKA-1164: - Fix Version/s: (was: 0.8.2) 0.8.1 > kafka should depend on snappy 1.0.5 (i

[jira] [Commented] (KAFKA-1232) make TopicCommand more consistent

2014-02-02 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13889030#comment-13889030 ] Jun Rao commented on KAFKA-1232: Updated reviewboard https://reviews.apache.org/r/17479/

[jira] [Updated] (KAFKA-1232) make TopicCommand more consistent

2014-02-02 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao updated KAFKA-1232: --- Attachment: KAFKA-1232_2014-02-02_11:24:42.patch > make TopicCommand more consistent > --

Re: Review Request 17479: Patch for KAFKA-1232

2014-02-02 Thread Jun Rao
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/17479/ --- (Updated Feb. 2, 2014, 7:24 p.m.) Review request for kafka. Bugs: KAFKA-1232

[jira] [Commented] (KAFKA-1164) kafka should depend on snappy 1.0.5 (instead of 1.0.4.1)

2014-02-02 Thread Jason Rosenberg (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13889027#comment-13889027 ] Jason Rosenberg commented on KAFKA-1164: Can this be reconsidered for 0.8.1? Seem

[jira] [Created] (KAFKA-1237) Add mirror maker using 08 consumer and 09 producer

2014-02-02 Thread Neha Narkhede (JIRA)
Neha Narkhede created KAFKA-1237: Summary: Add mirror maker using 08 consumer and 09 producer Key: KAFKA-1237 URL: https://issues.apache.org/jira/browse/KAFKA-1237 Project: Kafka Issue Type:

Re: Proposed Changes To New Producer Public API

2014-02-02 Thread Neha Narkhede
It *is* expected that the user call this method on every single call and because we no longer control the partitioner interface there will be no way to control this. Make sense. This will ensure that new partitions are detected as defined by topic.metadata.refresh.interval.ms. There are a quite

Re: Proposed Changes To New Producer Public API

2014-02-02 Thread Jay Kreps
Hey Neha, Basically partitionsForTopic will invoke Metadata.fetch(topic).partitionsFor. So it will block on the first request for a given topic while metadata is loaded. Each subsequent request will return whatever metadata is present, the logic for refreshing metadata will remain exactly as it is

Re: Proposed Changes To New Producer Public API

2014-02-02 Thread Jay Kreps
Hey Joel, I actually went with Future instead of Future but yes, as you say that object represents the topic, partition, and offset and would be the place we would put any future info we need to return to the user. The problem with implementing cancel is that the record is already serialized into