Re: Review Request 17263: New producer for Kafka.

> On Jan. 28, 2014, 8:48 p.m., Joel Koshy wrote: > > clients/src/main/java/kafka/clients/producer/KafkaProducer.java, line 182 > > > > > > This could add a couple minutes startup for producers that send to > > several

Re: New Producer Public API

I thought a bit about it and I think the getCluster() thing was overly simplistic because we try to only maintain metadata about the current set of topics the producer cares about so the cluster might not have the partitions for the topic the user cares about. I think actually what we need is a new

Re: New Producer Public API

+ dev (this thread has become a bit unwieldy) On Thu, Jan 30, 2014 at 5:15 PM, Joel Koshy wrote: > Does it preclude those various implementations? i.e., it could become > a producer config: > default.partitioner.strategy="minimize-connections"/"roundrobin" - and > so on; and implement those par

Re: New Producer Public API

That's a good point about 1A - does seem that we would need to have some kind of TTL for each topic's metadata. Also, WRT ZK dependency I don't think that decision (for the Java client) affects other clients. i.e., other client implementations can use whatever discovery mechanism it chooses. That

Re: New Producer Public API

With option 1A, if we increase # partitions on a topic, how will the producer find out newly created partitions? Do we expect the producer to periodically call getCluster()? As for ZK dependency, one of the goals of client rewrite is to reduce dependencies so that one can implement the client in l

Re: New Producer Public API

One downside to the 1A proposal is that without a Partitioner interface we can't really package up and provide common partitioner implementations. Example of these would be 1. HashPartitioner - The default hash partitioning 2. RoundRobinPartitioner - Just round-robins over partitions 3. ConnectionM

[jira] [Updated] (KAFKA-1028) per topic configuration of preference for consistency over availability

[ https://issues.apache.org/jira/browse/KAFKA-1028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Olson updated KAFKA-1028: Attachment: (was: KAFKA-1028.patch) > per topic configuration of preference for consistency ove

[jira] [Commented] (KAFKA-1028) per topic configuration of preference for consistency over availability

[ https://issues.apache.org/jira/browse/KAFKA-1028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13886980#comment-13886980 ] Andrew Olson commented on KAFKA-1028: - Updated reviewboard https://reviews.apache.org/

[jira] [Updated] (KAFKA-1028) per topic configuration of preference for consistency over availability

[ https://issues.apache.org/jira/browse/KAFKA-1028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Olson updated KAFKA-1028: Attachment: KAFKA-1028_2014-01-30_13:45:30.patch > per topic configuration of preference for consis

Re: Review Request 17537: Patch for KAFKA-1028

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/17537/ --- (Updated Jan. 30, 2014, 7:45 p.m.) Review request for kafka. Bugs: KAFKA-1028

[jira] [Updated] (KAFKA-1028) per topic configuration of preference for consistency over availability

[ https://issues.apache.org/jira/browse/KAFKA-1028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Olson updated KAFKA-1028: Attachment: (was: KAFKA-1028_2014-01-30_13:42:38.patch) > per topic configuration of preference

[jira] [Updated] (KAFKA-1028) per topic configuration of preference for consistency over availability

[ https://issues.apache.org/jira/browse/KAFKA-1028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Olson updated KAFKA-1028: Attachment: KAFKA-1028_2014-01-30_13:42:38.patch > per topic configuration of preference for consis

[jira] [Commented] (KAFKA-1028) per topic configuration of preference for consistency over availability

[ https://issues.apache.org/jira/browse/KAFKA-1028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13886976#comment-13886976 ] Andrew Olson commented on KAFKA-1028: - Updated reviewboard https://reviews.apache.org/

[jira] [Issue Comment Deleted] (KAFKA-1028) per topic configuration of preference for consistency over availability

[ https://issues.apache.org/jira/browse/KAFKA-1028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Olson updated KAFKA-1028: Comment: was deleted (was: Updated reviewboard https://reviews.apache.org/r/17537/ against branch

Re: Review Request 17537: Patch for KAFKA-1028

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/17537/ --- (Updated Jan. 30, 2014, 7:42 p.m.) Review request for kafka. Bugs: KAFKA-1028

[jira] Subscription: outstanding kafka patches

Issue Subscription Filter: outstanding kafka patches (85 issues) The list of outstanding kafka patches Subscriber: kafka-mailing-list Key Summary KAFKA-1233 Integration test for the new producer https://issues.apache.org/jira/browse/KAFKA-1233 KAFKA-1232 make TopicCommand mor

Re: Review Request 17263: New producer for Kafka.

> On Jan. 28, 2014, 8:48 p.m., Joel Koshy wrote: > > Overall, this looks great. Here are a few comments - I have some more > > minor/stylistic comments (typos in comments and broken javadoc, code > > conventions, minor javadoc edits and such) which I would rather defer for a > > later more final

[jira] [Updated] (KAFKA-1228) Socket Leak on ReplicaFetcherThread

[ https://issues.apache.org/jira/browse/KAFKA-1228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao updated KAFKA-1228: --- Resolution: Fixed Fix Version/s: 0.8.1 Assignee: Ahmy Yulrizka (was: Jun Rao) Stat

Re: Review Request 17263: New producer for Kafka.

Yeah I will definitely do this. One todo is to work out the final config names and document them all. -Jay On Wed, Jan 29, 2014 at 11:29 AM, Neha Narkhede wrote: > > > > On Jan. 26, 2014, 11:30 p.m., Sriram Subramanian wrote: > > > > clients/src/main/java/kafka/clients/producer/internals/Record

Re: Review Request 17524: Add one integration test for the new producer

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/17524/#review33232 --- Could you build a release tar and make sure that we don't drag in th

Re: Review Request 17479: Patch for KAFKA-1232

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/17479/#review33230 --- Ship it! Ship It! - Neha Narkhede On Jan. 30, 2014, 5:51 p.m., J

[jira] [Updated] (KAFKA-1232) make TopicCommand more consistent

[ https://issues.apache.org/jira/browse/KAFKA-1232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao updated KAFKA-1232: --- Attachment: KAFKA-1232_2014-01-30_09:51:48.patch > make TopicCommand more consistent > --

[jira] [Commented] (KAFKA-1232) make TopicCommand more consistent

[ https://issues.apache.org/jira/browse/KAFKA-1232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13886819#comment-13886819 ] Jun Rao commented on KAFKA-1232: Updated reviewboard https://reviews.apache.org/r/17479/

Re: Review Request 17479: Patch for KAFKA-1232

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/17479/ --- (Updated Jan. 30, 2014, 5:51 p.m.) Review request for kafka. Bugs: KAFKA-1232

Re: Review Request 17524: Add one integration test for the new producer

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/17524/#review33225 --- Ship it! This is a very nice test! It would be good to also have a

Re: Published Java/Scala docs?

Yeah perhaps we should do that. The problem we had before was that we hadn't seperated the user-facing versus non-user-facing classes so there were like 10k scala classes. It was pretty overwhelming. -Jay On Thu, Jan 30, 2014 at 4:55 AM, David Arthur wrote: > Yea, I thought the Scaladocs for 0

[jira] [Updated] (KAFKA-1184) High-Level Consumer: expose fetcher threads number as a parameter

[ https://issues.apache.org/jira/browse/KAFKA-1184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede updated KAFKA-1184: - Resolution: Fixed Status: Resolved (was: Patch Available) Committed to trunk > High-Lev

[jira] [Closed] (KAFKA-1184) High-Level Consumer: expose fetcher threads number as a parameter

[ https://issues.apache.org/jira/browse/KAFKA-1184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede closed KAFKA-1184. > High-Level Consumer: expose fetcher threads number as a parameter > -

[jira] [Commented] (KAFKA-1184) High-Level Consumer: expose fetcher threads number as a parameter

[ https://issues.apache.org/jira/browse/KAFKA-1184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13886752#comment-13886752 ] Neha Narkhede commented on KAFKA-1184: -- Thanks for the patch. Overall, looks good. Ma

[jira] [Issue Comment Deleted] (KAFKA-1028) per topic configuration of preference for consistency over availability

[ https://issues.apache.org/jira/browse/KAFKA-1028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Olson updated KAFKA-1028: Comment: was deleted (was: Created reviewboard against branch origin/trunk) > per topic configura

[jira] [Updated] (KAFKA-1028) per topic configuration of preference for consistency over availability

[ https://issues.apache.org/jira/browse/KAFKA-1028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Olson updated KAFKA-1028: Attachment: (was: KAFKA-1028.patch) > per topic configuration of preference for consistency ove

[jira] [Commented] (KAFKA-1028) per topic configuration of preference for consistency over availability

[ https://issues.apache.org/jira/browse/KAFKA-1028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13886644#comment-13886644 ] Andrew Olson commented on KAFKA-1028: - Reviewboard has been created. I'll make the sug

[jira] [Commented] (KAFKA-1028) per topic configuration of preference for consistency over availability

[ https://issues.apache.org/jira/browse/KAFKA-1028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13886637#comment-13886637 ] Andrew Olson commented on KAFKA-1028: - Created reviewboard https://reviews.apache.org/

[jira] [Updated] (KAFKA-1028) per topic configuration of preference for consistency over availability

[ https://issues.apache.org/jira/browse/KAFKA-1028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Olson updated KAFKA-1028: Attachment: KAFKA-1028.patch > per topic configuration of preference for consistency over availabil

Review Request 17537: Patch for KAFKA-1028

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/17537/ --- Review request for kafka. Bugs: KAFKA-1028 https://issues.apache.org/jira/b

[jira] [Commented] (KAFKA-1028) per topic configuration of preference for consistency over availability

[ https://issues.apache.org/jira/browse/KAFKA-1028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13886635#comment-13886635 ] Andrew Olson commented on KAFKA-1028: - Created reviewboard against branch origin/trun

[jira] [Updated] (KAFKA-1028) per topic configuration of preference for consistency over availability

[ https://issues.apache.org/jira/browse/KAFKA-1028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Olson updated KAFKA-1028: Attachment: KAFKA-1028.patch > per topic configuration of preference for consistency over availabil

Re: Published Java/Scala docs?

Yea, I thought the Scaladocs for 0.7 were perfectly fine. Even with Javadocs for the new clients won't help with lower level classes that are exposed in those APIs (e.g., KeyedMessage) On 1/30/14 6:26 AM, Kostya Golikov wrote: Why not use *scala*doc then? It is way prettier than javadoc 2014

[jira] [Updated] (KAFKA-1184) High-Level Consumer: expose fetcher threads number as a parameter

[ https://issues.apache.org/jira/browse/KAFKA-1184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Evelina Stepanova updated KAFKA-1184: - Attachment: configurable-consumer-fetch-threads.patch > High-Level Consumer: expose fetch

[jira] [Updated] (KAFKA-1184) High-Level Consumer: expose fetcher threads number as a parameter

[ https://issues.apache.org/jira/browse/KAFKA-1184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Evelina Stepanova updated KAFKA-1184: - Labels: patch (was: ) Status: Patch Available (was: Open) * Number of fetcher threa

Re: Published Java/Scala docs?

Why not use *scala*doc then? It is way prettier than javadoc 2014-01-30 Jay Kreps : > The primary problem was that we never found a way to get sane javadocs for > scala code. Moving the clients out of scala will fix this (e.g. > > http://empathybox.com/kafka-javadoc/index.html?kafka/clients/prod

[jira] [Commented] (KAFKA-260) Add audit trail to kafka

[ https://issues.apache.org/jira/browse/KAFKA-260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13886455#comment-13886455 ] Daniel Tardón commented on KAFKA-260: - Hi all, Ashok, do you have any news about this