[jira] [Created] (KAFKA-2170) 10 LogTest cases failed for file.renameTo failed under windows

Honghai Chen created KAFKA-2170: --- Summary: 10 LogTest cases failed for file.renameTo failed under windows Key: KAFKA-2170 URL: https://issues.apache.org/jira/browse/KAFKA-2170 Project: Kafka

RE: [DISCUSS] KIP 20 Enable log preallocate to improve consume performance under windows and some old Linux file system

Use config.segmentSize should be ok. Previously add that one for make sure the file not exceed config.segmentSize, actually the function maybeRoll already make sure that. When try add test case for recover, blocked by the rename related issue, just open one jira at https://issues.apache.org/ji

[jira] [Commented] (KAFKA-2156) Possibility to plug in custom MetricRegistry

[ https://issues.apache.org/jira/browse/KAFKA-2156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14528203#comment-14528203 ] Andras Sereny commented on KAFKA-2156: -- Thanks Neha. This might already work with Kaf

[jira] [Comment Edited] (KAFKA-2156) Possibility to plug in custom MetricRegistry

[ https://issues.apache.org/jira/browse/KAFKA-2156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14528203#comment-14528203 ] Andras Sereny edited comment on KAFKA-2156 at 5/5/15 10:58 AM: -

Re: [DISCUSS] Using GitHub Pull Requests for contributions and code review

Hi Ewen, Comments inline. On Fri, May 1, 2015 at 8:38 AM, Ewen Cheslack-Postava wrote: > Also +1. There are some drawbacks to using Github for reviews, e.g. lots of > emails for each review because they don't let you publish your entire > review in one go like RB does, but it drastically lowers

Re: [DISCUSS] Using GitHub Pull Requests for contributions and code review

Hi Jun, On Sat, May 2, 2015 at 2:50 PM, Jun Rao wrote: > We will also need to figure out if we need CONTRIBUTING.md like the > following to take care of the Apache licensing stuff. > > https://github.com/apache/spark/blob/master/CONTRIBUTING.md Yes indeed. That is in step 3 of the "missing pie

Re: [KIP-DISCUSSION] KIP-22 Expose a Partitioner interface in the new producer

Thanks for the comments everyone. Hi Jay,      I do have a question regarding configurable interface on how to pass a Map properties. I couldn’t find any other classes using it. JMX reporter overrides it but doesn’t implement it.  So with configurable partitioner how can a user pass in partition

Re: Kafka KIP hangout May 5

Is there information on how to get into the hangout?  Or is it by invitation?   I'm very interested in the authorization changes. Thanks,Tom On Monday, May 4, 2015 9:20 AM, Jun Rao wrote: Hi, Everyone, We will have a KIP hangout at 11 PST on May 5. The following is the agenda. If yo

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations (Thread 2)

Guys, I've updated the wiki to reflect all previously discussed items (regarding the schema only - this is included to phase 1). https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations I think we can have the final discussion today (for pha

Re: [DISCUSS] Using GitHub Pull Requests for contributions and code review

Yes, it would be great if we can do a final rebase of the branch before merging. Perhaps the squashing part already does that? Thanks, Jun On Tue, May 5, 2015 at 5:53 AM, Ismael Juma wrote: > Hi Jun, > > On Sat, May 2, 2015 at 2:50 PM, Jun Rao wrote: > > > We will also need to figure out if w

[jira] [Updated] (KAFKA-1387) Kafka getting stuck creating ephemeral node it has already created when two zookeeper sessions are established in a very short period of time

[ https://issues.apache.org/jira/browse/KAFKA-1387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao updated KAFKA-1387: --- Reviewer: Jun Rao > Kafka getting stuck creating ephemeral node it has already created when two > zookeeper s

[jira] [Updated] (KAFKA-2165) ReplicaFetcherThread: data loss on unknown exception

[ https://issues.apache.org/jira/browse/KAFKA-2165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao updated KAFKA-2165: --- Reviewer: Jun Rao > ReplicaFetcherThread: data loss on unknown exception > ---

[jira] [Updated] (KAFKA-1928) Move kafka.network over to using the network classes in org.apache.kafka.common.network

[ https://issues.apache.org/jira/browse/KAFKA-1928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao updated KAFKA-1928: --- Reviewer: Jay Kreps > Move kafka.network over to using the network classes in > org.apache.kafka.common.netwo

Re: Review Request 33760: Patch for KAFKA-2121

> On May 4, 2015, 5:31 p.m., Neha Narkhede wrote: > > clients/src/main/java/org/apache/kafka/common/serialization/Serializer.java, > > line 25 > > > > > > This and also the Deserializer should extend Configurable too r

Re: [DISCUSS] KIP-21 Configuration Management

Good discussion. Since we will be talking about this at 11am, I wanted to organize these comments into requirements to see if we are all on the same page. REQUIREMENT 1: Needs to accept dynamic config changes. This needs to be general enough to work for all configs that we envision may need to acc

[jira] [Updated] (KAFKA-2160) DelayedOperationPurgatory should remove the pair in watchersForKey with empty watchers list

[ https://issues.apache.org/jira/browse/KAFKA-2160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao updated KAFKA-2160: --- Reviewer: Jun Rao > DelayedOperationPurgatory should remove the pair in watchersForKey with empty > watchers

[jira] [Updated] (KAFKA-2161) Fix a few copyrights

[ https://issues.apache.org/jira/browse/KAFKA-2161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao updated KAFKA-2161: --- Reviewer: Joe Stein > Fix a few copyrights > > > Key: KAFKA-2161 >

[jira] [Updated] (KAFKA-2129) Consumer could make multiple concurrent metadata requests

[ https://issues.apache.org/jira/browse/KAFKA-2129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao updated KAFKA-2129: --- Reviewer: Guozhang Wang > Consumer could make multiple concurrent metadata requests >

[jira] [Updated] (KAFKA-1690) new java producer needs ssl support as a client

[ https://issues.apache.org/jira/browse/KAFKA-1690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao updated KAFKA-1690: --- Reviewer: Jun Rao > new java producer needs ssl support as a client >

[jira] [Updated] (KAFKA-2123) Make new consumer offset commit API use callback + future

[ https://issues.apache.org/jira/browse/KAFKA-2123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao updated KAFKA-2123: --- Reviewer: Jay Kreps > Make new consumer offset commit API use callback + future >

[jira] [Updated] (KAFKA-2121) prevent potential resource leak in KafkaProducer and KafkaConsumer

[ https://issues.apache.org/jira/browse/KAFKA-2121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao updated KAFKA-2121: --- Reviewer: Guozhang Wang > prevent potential resource leak in KafkaProducer and KafkaConsumer > ---

Re: Adding multi-tenancy capabilities to Kafka

Adrian, Trying to follow up the discussion here. Is my understanding correct that if we have topic hierarchies then we can do without namespaces. To me namespace is an abstraction, it can be implemented with topic hierarchies as well, would you agree? If so I guess topic hierarchies is the way to

[jira] [Created] (KAFKA-2171) System Test for Quotas

Dong Lin created KAFKA-2171: --- Summary: System Test for Quotas Key: KAFKA-2171 URL: https://issues.apache.org/jira/browse/KAFKA-2171 Project: Kafka Issue Type: Sub-task Reporter: Dong Li

Re: Review Request 33088: add heartbeat to coordinator

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33088/ --- (Updated May 5, 2015, 5:50 p.m.) Review request for kafka. Bugs: KAFKA-1334

[jira] [Commented] (KAFKA-1334) Add failure detection capability to the coordinator / consumer

[ https://issues.apache.org/jira/browse/KAFKA-1334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14528896#comment-14528896 ] Onur Karaman commented on KAFKA-1334: - Updated reviewboard https://reviews.apache.org/

[jira] [Updated] (KAFKA-1334) Add failure detection capability to the coordinator / consumer

[ https://issues.apache.org/jira/browse/KAFKA-1334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Onur Karaman updated KAFKA-1334: Attachment: KAFKA-1334_2015-05-05_10:50:00.patch > Add failure detection capability to the coordinat

Re: Review Request 33088: add heartbeat to coordinator

> On April 28, 2015, 12:13 a.m., Guozhang Wang wrote: > > clients/src/main/java/org/apache/kafka/common/protocol/Errors.java, lines > > 73-76 > > > > > > Maybe we can merge them into one error code, > > UNKNOWN_OR_INC

[VOTE] KIP-4 Admin Commands / Phase-1

Hi all, This is a voting thread for KIP-4 Phase-1. It will include Wire protocol changes and server side handling code. https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations Thanks, Andrii Biletskyi

Re: Adding multi-tenancy capabilities to Kafka

Hey guys, It would be nice to get a design around this. Though there are currently so many big things in flight I do wonder if we should start another parallel thing...? But working out a design can't hurt. Personally I think since one of the goals of Kafka is data integration we really want to s

Re: [DISCUSS] KIP-21 Configuration Management

Joel, thanks for summarizing the requirements. It makes sense for the KIP to focus on Req #1, unless any future configs as dynamic ones warrants a completely different design. My main concern is going with a design by keeping only quotas in mind and then continue shoehorning other dynamic configs i

Review Request 33857: Patch for KAFKA-2103

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33857/ --- Review request for kafka. Bugs: KAFKA-2103 https://issues.apache.org/jira/b

[jira] [Updated] (KAFKA-2103) kafka.producer.AsyncProducerTest failure.

[ https://issues.apache.org/jira/browse/KAFKA-2103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ewen Cheslack-Postava updated KAFKA-2103: - Attachment: KAFKA-2103.patch > kafka.producer.AsyncProducerTest failure. > ---

[jira] [Commented] (KAFKA-2103) kafka.producer.AsyncProducerTest failure.

[ https://issues.apache.org/jira/browse/KAFKA-2103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14529033#comment-14529033 ] Ewen Cheslack-Postava commented on KAFKA-2103: -- Agreed with [~becket_qin] tha

[jira] [Updated] (KAFKA-2103) kafka.producer.AsyncProducerTest failure.

[ https://issues.apache.org/jira/browse/KAFKA-2103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ewen Cheslack-Postava updated KAFKA-2103: - Assignee: Ewen Cheslack-Postava (was: Dong Lin) Status: Patch Available (wa

[jira] [Commented] (KAFKA-2103) kafka.producer.AsyncProducerTest failure.

[ https://issues.apache.org/jira/browse/KAFKA-2103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14529032#comment-14529032 ] Ewen Cheslack-Postava commented on KAFKA-2103: -- Created reviewboard https://r

[jira] [Commented] (KAFKA-1690) new java producer needs ssl support as a client

[ https://issues.apache.org/jira/browse/KAFKA-1690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14529043#comment-14529043 ] Sriharsha Chintalapani commented on KAFKA-1690: --- [~junrao] [~jjkoshy] I got

Re: Adding multi-tenancy capabilities to Kafka

Jay, I agree with you. This will kind of depend on how KIP-11 and KIP-12 shapes up, but we can definitely start putting together the ideas. Would you suggest starting a kip or just a generic shared document that can be later translated into a KIP? On Tuesday, May 5, 2015, Jay Kreps wrote: > Hey

Re: Adding multi-tenancy capabilities to Kafka

I agree. If we can really do pluggable authorization (and even pluggable authentication), it would not actually be hard to effectively implement a multi-tenant solution. I am hoping to attempt to implement something like this once there is code / patches for KIP-11 and KIP-12. So I wonder if we

Re: Adding multi-tenancy capabilities to Kafka

I'm don't agree. At least not completely ;) Pluggable authorization (or hierarchies with the ability to define ACLs at each level) would allow you to isolate different groups of users. However each user will still be "aware" of one another's presence in the topic namespace - and could also b

Re: Adding multi-tenancy capabilities to Kafka

I'm don't agree. At least not completely ;) Pluggable authorization (or hierarchies with the ability to define ACLs at each level) would allow you to isolate different groups of users. However each user will still be "aware" of one another's presence in the topic namespace - and could also be

[DISCUSSION] Reuse o.a.k.clients.NetworkClient in controller.

I am trying to see if we can reuse the NetworkClient class to be used in controller to broker communication. (Also, we can probably use KafkaConsumer which is already using NetworkClient in replica fetchers). Currently NetworkClient does the following things in addition to sending requests. 1

[jira] [Commented] (KAFKA-2170) 10 LogTest cases failed for file.renameTo failed under windows

[ https://issues.apache.org/jira/browse/KAFKA-2170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14529337#comment-14529337 ] Jun Rao commented on KAFKA-2170: We do have the logic in OffsetIndex.resize() to unmap bef

Re: [DISCUSS] KIP 20 Enable log preallocate to improve consume performance under windows and some old Linux file system

Thanks. Could you updated the wiki? Also, commented on the jira. Jun On Tue, May 5, 2015 at 12:48 AM, Honghai Chen wrote: > Use config.segmentSize should be ok. Previously add that one for make > sure the file not exceed config.segmentSize, actually the function > maybeRoll already make sure

Re: Review Request 33088: add heartbeat to coordinator

> On April 28, 2015, 12:13 a.m., Guozhang Wang wrote: > > core/src/main/scala/kafka/coordinator/ConsumerCoordinator.scala, lines > > 281-355 > > > > > > If we add the CoordinatorMetadata class, these functions can all

Re: Review Request 33049: Patch for KAFKA-2084

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33049/ --- (Updated May 5, 2015, 10:27 p.m.) Review request for kafka, Joel Koshy and Jun

[jira] [Commented] (KAFKA-2084) byte rate metrics per client ID (producer and consumer)

[ https://issues.apache.org/jira/browse/KAFKA-2084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14529422#comment-14529422 ] Aditya A Auradkar commented on KAFKA-2084: -- Updated reviewboard https://reviews.a

[jira] [Updated] (KAFKA-2084) byte rate metrics per client ID (producer and consumer)

[ https://issues.apache.org/jira/browse/KAFKA-2084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aditya A Auradkar updated KAFKA-2084: - Attachment: KAFKA-2084_2015-05-05_15:27:35.patch > byte rate metrics per client ID (produc

Re: Review Request 33049: Patch for KAFKA-2084

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33049/ --- (Updated May 5, 2015, 10:29 p.m.) Review request for kafka, Joel Koshy and Jun

[jira] [Created] (KAFKA-2172) Round-robin partition assignment strategy too restrictive

Jason Rosenberg created KAFKA-2172: -- Summary: Round-robin partition assignment strategy too restrictive Key: KAFKA-2172 URL: https://issues.apache.org/jira/browse/KAFKA-2172 Project: Kafka I

[jira] [Commented] (KAFKA-2172) Round-robin partition assignment strategy too restrictive

[ https://issues.apache.org/jira/browse/KAFKA-2172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14529436#comment-14529436 ] Jason Rosenberg commented on KAFKA-2172: as a side note, the documentation here:

Re: [DISCUSS] KIP-21 Configuration Management

Sorry I missed the call today :) I think an additional requirement would be: Make sure that traditional deployment tools (Puppet, Chef, etc) are still capable of managing Kafka configuration. For this reason, I'd like the configuration refresh to be pretty close to what most Linux services are do

Re: [DISCUSSION] Reuse o.a.k.clients.NetworkClient in controller.

Hi, Jiangjie, Thanks for taking on this. I was thinking that one way to decouple the dependency on Metadata in NetworkClient is the following. 1. Make Metadata an interface. 2. Rename current Metadata class to sth like KafkaMetadata that implements the Metadata interface. 3. Have a new NoOpMetada

Re: Kafka KIP hangout May 5

Below are my notes from today’s KIP hangout. Please feel free to add/ correct the content. Kafka KIP Discussion (May 5) KIP-4 (admin commands): any remaining issues - Only issue left is how to handle multiple instructions for one topic. Suggestion here was to silently ignore duplicate instru

[jira] [Commented] (KAFKA-1737) Document required ZkSerializer for ZkClient used with AdminUtils

[ https://issues.apache.org/jira/browse/KAFKA-1737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14529638#comment-14529638 ] Vivek Madani commented on KAFKA-1737: - Sure - will rebase and upload > Document requi

Re: Review Request 33049: Patch for KAFKA-2084

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33049/ --- (Updated May 6, 2015, 12:52 a.m.) Review request for kafka, Joel Koshy and Jun

[jira] [Commented] (KAFKA-2084) byte rate metrics per client ID (producer and consumer)

[ https://issues.apache.org/jira/browse/KAFKA-2084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14529672#comment-14529672 ] Aditya A Auradkar commented on KAFKA-2084: -- Updated reviewboard https://reviews.a

Re: Review Request 33049: Patch for KAFKA-2084

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33049/ --- (Updated May 6, 2015, 12:52 a.m.) Review request for kafka, Joel Koshy and Jun

[jira] [Updated] (KAFKA-2084) byte rate metrics per client ID (producer and consumer)

[ https://issues.apache.org/jira/browse/KAFKA-2084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aditya A Auradkar updated KAFKA-2084: - Attachment: KAFKA-2084_2015-05-05_17:52:02.patch > byte rate metrics per client ID (produc

[jira] [Commented] (KAFKA-1646) Improve consumer read performance for Windows

[ https://issues.apache.org/jira/browse/KAFKA-1646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14529694#comment-14529694 ] Honghai Chen commented on KAFKA-1646: - When trying add test case for Log, got test fai

[jira] [Comment Edited] (KAFKA-1646) Improve consumer read performance for Windows

[ https://issues.apache.org/jira/browse/KAFKA-1646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14529694#comment-14529694 ] Honghai Chen edited comment on KAFKA-1646 at 5/6/15 1:13 AM: -

Re: [DISCUSS] KIP-19 Add a request timeout to NetworkClient

Just a quick question, can we handle REQUEST TIMEOUT as disconnections and do a fresh MetaDataRequest and retry instead of failing the request? Thanks, Mayuresh On Mon, May 4, 2015 at 10:32 AM, Jiangjie Qin wrote: > I incorporated Ewen and Guozhang’s comments in the KIP page. Want to speed >

Re: [DISCUSSION] Reuse o.a.k.clients.NetworkClient in controller.

+1 on trying to reuse the NetworkClient code. I think Jun's approach could work, but I'm wondering if refactoring a bit could get better separation of concerns without a somewhat awkward nop implementation of Metadata. I'm not sure what combination of delegation or subclassing makes sense yet, but