[jira] [Created] (KAFKA-2310) Add config to prevent broker becoming controller

2015-07-06 Thread Andrii Biletskyi (JIRA)
Andrii Biletskyi created KAFKA-2310:
---

 Summary: Add config to prevent broker becoming controller
 Key: KAFKA-2310
 URL: https://issues.apache.org/jira/browse/KAFKA-2310
 Project: Kafka
  Issue Type: Bug
Reporter: Andrii Biletskyi
Assignee: Andrii Biletskyi


The goal is to be able to specify which cluster brokers can serve as a 
controller and which cannot. This way it will be possible to "reserve" 
particular, not overloaded with partitions and other operations, broker as 
controller.

Proposed to add config _controller.eligibility_ defaulted to true (for backward 
compatibility, since now any broker can become a controller)

Patch will be available for trunk, 0.8.2 and 0.8.1



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


Review Request 36203: Patch for KAFKA-2310

2015-07-06 Thread Andrii Biletskyi

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36203/
---

Review request for kafka.


Bugs: KAFKA-2310
https://issues.apache.org/jira/browse/KAFKA-2310


Repository: kafka


Description
---

KAFKA-2310 - Add config to prevent broker becoming controller


Diffs
-

  core/src/main/scala/kafka/controller/KafkaController.scala 
36350579b16027359d237b64699003358704ac6f 
  core/src/main/scala/kafka/server/KafkaConfig.scala 
c1f0ccad4900d74e41936fae4c6c2235fe9314fe 
  core/src/main/scala/kafka/server/NoOpLeaderElector.scala PRE-CREATION 
  core/src/test/scala/unit/kafka/server/KafkaConfigConfigDefTest.scala 
98a5b042a710d3c1064b0379db1d152efc9eabee 

Diff: https://reviews.apache.org/r/36203/diff/


Testing
---


Thanks,

Andrii Biletskyi



[jira] [Commented] (KAFKA-2310) Add config to prevent broker becoming controller

2015-07-06 Thread Andrii Biletskyi (JIRA)

[ 
https://issues.apache.org/jira/browse/KAFKA-2310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14615111#comment-14615111
 ] 

Andrii Biletskyi commented on KAFKA-2310:
-

Created reviewboard https://reviews.apache.org/r/36203/diff/
 against branch origin/trunk

> Add config to prevent broker becoming controller
> 
>
> Key: KAFKA-2310
> URL: https://issues.apache.org/jira/browse/KAFKA-2310
> Project: Kafka
>  Issue Type: Bug
>Reporter: Andrii Biletskyi
>Assignee: Andrii Biletskyi
> Attachments: KAFKA-2310.patch
>
>
> The goal is to be able to specify which cluster brokers can serve as a 
> controller and which cannot. This way it will be possible to "reserve" 
> particular, not overloaded with partitions and other operations, broker as 
> controller.
> Proposed to add config _controller.eligibility_ defaulted to true (for 
> backward compatibility, since now any broker can become a controller)
> Patch will be available for trunk, 0.8.2 and 0.8.1



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (KAFKA-2310) Add config to prevent broker becoming controller

2015-07-06 Thread Andrii Biletskyi (JIRA)

 [ 
https://issues.apache.org/jira/browse/KAFKA-2310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrii Biletskyi updated KAFKA-2310:

Attachment: KAFKA-2310.patch

> Add config to prevent broker becoming controller
> 
>
> Key: KAFKA-2310
> URL: https://issues.apache.org/jira/browse/KAFKA-2310
> Project: Kafka
>  Issue Type: Bug
>Reporter: Andrii Biletskyi
>Assignee: Andrii Biletskyi
> Attachments: KAFKA-2310.patch
>
>
> The goal is to be able to specify which cluster brokers can serve as a 
> controller and which cannot. This way it will be possible to "reserve" 
> particular, not overloaded with partitions and other operations, broker as 
> controller.
> Proposed to add config _controller.eligibility_ defaulted to true (for 
> backward compatibility, since now any broker can become a controller)
> Patch will be available for trunk, 0.8.2 and 0.8.1



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (KAFKA-2310) Add config to prevent broker becoming controller

2015-07-06 Thread Andrii Biletskyi (JIRA)

 [ 
https://issues.apache.org/jira/browse/KAFKA-2310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrii Biletskyi updated KAFKA-2310:

Status: Patch Available  (was: Open)

> Add config to prevent broker becoming controller
> 
>
> Key: KAFKA-2310
> URL: https://issues.apache.org/jira/browse/KAFKA-2310
> Project: Kafka
>  Issue Type: Bug
>Reporter: Andrii Biletskyi
>Assignee: Andrii Biletskyi
> Attachments: KAFKA-2310.patch
>
>
> The goal is to be able to specify which cluster brokers can serve as a 
> controller and which cannot. This way it will be possible to "reserve" 
> particular, not overloaded with partitions and other operations, broker as 
> controller.
> Proposed to add config _controller.eligibility_ defaulted to true (for 
> backward compatibility, since now any broker can become a controller)
> Patch will be available for trunk, 0.8.2 and 0.8.1



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (KAFKA-2310) Add config to prevent broker becoming controller

2015-07-06 Thread Andrii Biletskyi (JIRA)

 [ 
https://issues.apache.org/jira/browse/KAFKA-2310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrii Biletskyi updated KAFKA-2310:

Attachment: KAFKA-2310_0.8.2.patch

> Add config to prevent broker becoming controller
> 
>
> Key: KAFKA-2310
> URL: https://issues.apache.org/jira/browse/KAFKA-2310
> Project: Kafka
>  Issue Type: Bug
>Reporter: Andrii Biletskyi
>Assignee: Andrii Biletskyi
> Attachments: KAFKA-2310.patch, KAFKA-2310_0.8.2.patch
>
>
> The goal is to be able to specify which cluster brokers can serve as a 
> controller and which cannot. This way it will be possible to "reserve" 
> particular, not overloaded with partitions and other operations, broker as 
> controller.
> Proposed to add config _controller.eligibility_ defaulted to true (for 
> backward compatibility, since now any broker can become a controller)
> Patch will be available for trunk, 0.8.2 and 0.8.1



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


Re: Review Request 35880: Patch for KAFKA-2295

2015-07-06 Thread Guozhang Wang

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/35880/#review90488
---



clients/src/main/java/org/apache/kafka/common/config/ConfigDef.java (line 266)


Can we use Utils.newInstance here as well?



clients/src/test/java/org/apache/kafka/common/config/AbstractConfigTest.java 
(line 57)


Is this intentional?



core/src/main/scala/kafka/utils/CoreUtils.scala (line 221)


Can we use scala's Utils.createObject here?


- Guozhang Wang


On July 6, 2015, 6:05 a.m., Manikumar Reddy O wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/35880/
> ---
> 
> (Updated July 6, 2015, 6:05 a.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-2295
> https://issues.apache.org/jira/browse/KAFKA-2295
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> Addessing Guozhang's comments
> 
> 
> Diffs
> -
> 
>   clients/src/main/java/org/apache/kafka/common/config/AbstractConfig.java 
> bae528d31516679bed88ee61b408f209f185a8cc 
>   clients/src/main/java/org/apache/kafka/common/config/ConfigDef.java 
> 4170bcc7def5b50d8aa20e8e84089c35b705b527 
>   clients/src/main/java/org/apache/kafka/common/utils/Utils.java 
> af9993cf9b3991f1e61e1201c94e19bc1bf76a68 
>   
> clients/src/test/java/org/apache/kafka/common/config/AbstractConfigTest.java 
> db1b0ee9113215b5ad7fda0f93915f3bdd34ac55 
>   core/src/main/scala/kafka/utils/CoreUtils.scala 
> 168a18d380c200ee566eccb6988dd1ae85ed5b09 
> 
> Diff: https://reviews.apache.org/r/35880/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Manikumar Reddy O
> 
>



Re: Guozhang Wang elected to Kafka PMC

2015-07-06 Thread Jarek Jarcec Cecho
Congratulations Guozhang!

Jarcec

> On Jun 15, 2015, at 9:59 PM, Jun Rao  wrote:
> 
> Hi, Everyone,
> 
> Guozhang Wang has been active in the Kafka community since he became a
> Kafka committer about 7 months ago. I am glad to announce that Guozhang is
> now a member of Kafka PMC.
> 
> Congratulations, Guozhang!
> 
> Jun



[jira] [Reopened] (KAFKA-2199) Make signing artifacts optional, setting maven repository possible from command line

2015-07-06 Thread Ewen Cheslack-Postava (JIRA)

 [ 
https://issues.apache.org/jira/browse/KAFKA-2199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ewen Cheslack-Postava reopened KAFKA-2199:
--

Needs followup patch applied.

> Make signing artifacts optional, setting maven repository possible from 
> command line
> 
>
> Key: KAFKA-2199
> URL: https://issues.apache.org/jira/browse/KAFKA-2199
> Project: Kafka
>  Issue Type: Bug
>  Components: build
>Reporter: Ewen Cheslack-Postava
>Assignee: Ewen Cheslack-Postava
>Priority: Minor
> Attachments: KAFKA-2199.patch, KAFKA-2199.patch, 
> KAFKA-2199_2015-05-29_11:00:44.patch
>
>
> Currently it's annoying to work with snapshot builds if you want to install 
> them rather than just build & test. There are a couple of problems. First, if 
> you try to install (any of the upload* tasks), then you are required to sign 
> the artifacts with a GPG key. Second, the way the variables are defined in 
> gradle.properties seems to make it impossible to override them from the 
> command line. You're required to edit your ~/.gradle/gradle.properties file 
> (which is shared by all gradle projects).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (KAFKA-2310) Add config to prevent broker becoming controller

2015-07-06 Thread Andrii Biletskyi (JIRA)

 [ 
https://issues.apache.org/jira/browse/KAFKA-2310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrii Biletskyi updated KAFKA-2310:

Attachment: KAFKA-2310_0.8.1.patch

> Add config to prevent broker becoming controller
> 
>
> Key: KAFKA-2310
> URL: https://issues.apache.org/jira/browse/KAFKA-2310
> Project: Kafka
>  Issue Type: Bug
>Reporter: Andrii Biletskyi
>Assignee: Andrii Biletskyi
> Attachments: KAFKA-2310.patch, KAFKA-2310_0.8.1.patch, 
> KAFKA-2310_0.8.2.patch
>
>
> The goal is to be able to specify which cluster brokers can serve as a 
> controller and which cannot. This way it will be possible to "reserve" 
> particular, not overloaded with partitions and other operations, broker as 
> controller.
> Proposed to add config _controller.eligibility_ defaulted to true (for 
> backward compatibility, since now any broker can become a controller)
> Patch will be available for trunk, 0.8.2 and 0.8.1



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (KAFKA-2310) Add config to prevent broker becoming controller

2015-07-06 Thread Andrii Biletskyi (JIRA)

 [ 
https://issues.apache.org/jira/browse/KAFKA-2310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrii Biletskyi updated KAFKA-2310:

Attachment: (was: KAFKA-2310_0.8.2.patch)

> Add config to prevent broker becoming controller
> 
>
> Key: KAFKA-2310
> URL: https://issues.apache.org/jira/browse/KAFKA-2310
> Project: Kafka
>  Issue Type: Bug
>Reporter: Andrii Biletskyi
>Assignee: Andrii Biletskyi
> Attachments: KAFKA-2310.patch, KAFKA-2310_0.8.1.patch, 
> KAFKA-2310_0.8.2.patch
>
>
> The goal is to be able to specify which cluster brokers can serve as a 
> controller and which cannot. This way it will be possible to "reserve" 
> particular, not overloaded with partitions and other operations, broker as 
> controller.
> Proposed to add config _controller.eligibility_ defaulted to true (for 
> backward compatibility, since now any broker can become a controller)
> Patch will be available for trunk, 0.8.2 and 0.8.1



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (KAFKA-2310) Add config to prevent broker becoming controller

2015-07-06 Thread Andrii Biletskyi (JIRA)

 [ 
https://issues.apache.org/jira/browse/KAFKA-2310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrii Biletskyi updated KAFKA-2310:

Attachment: KAFKA-2310_0.8.2.patch

> Add config to prevent broker becoming controller
> 
>
> Key: KAFKA-2310
> URL: https://issues.apache.org/jira/browse/KAFKA-2310
> Project: Kafka
>  Issue Type: Bug
>Reporter: Andrii Biletskyi
>Assignee: Andrii Biletskyi
> Attachments: KAFKA-2310.patch, KAFKA-2310_0.8.1.patch, 
> KAFKA-2310_0.8.2.patch
>
>
> The goal is to be able to specify which cluster brokers can serve as a 
> controller and which cannot. This way it will be possible to "reserve" 
> particular, not overloaded with partitions and other operations, broker as 
> controller.
> Proposed to add config _controller.eligibility_ defaulted to true (for 
> backward compatibility, since now any broker can become a controller)
> Patch will be available for trunk, 0.8.2 and 0.8.1



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (KAFKA-2298) Client Selector can drop connections on InvalidReceiveException without notifying NetworkClient

2015-07-06 Thread Dong Lin (JIRA)

[ 
https://issues.apache.org/jira/browse/KAFKA-2298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14615622#comment-14615622
 ] 

Dong Lin commented on KAFKA-2298:
-

Joel, could you please review this patch when you have time?

> Client Selector can drop connections on InvalidReceiveException without 
> notifying NetworkClient
> ---
>
> Key: KAFKA-2298
> URL: https://issues.apache.org/jira/browse/KAFKA-2298
> Project: Kafka
>  Issue Type: Bug
>Reporter: Dong Lin
>Assignee: Dong Lin
>  Labels: quotas
> Attachments: KAFKA-2298.patch, KAFKA-2298_2015-06-23_18:47:54.patch, 
> KAFKA-2298_2015-06-24_13:00:39.patch
>
>
> I run into the problem described in KAFKA-2266 when testing quota. I was told 
> the bug was fixed in KAFKA-2266 after I figured out the problem.
> But the patch provided in KAFKA-2266 probably doesn't solve all related 
> problems. From reading the code there is still one edge case where the client 
> selector can close connection in poll() without notifying NetworkClient.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (KAFKA-2241) AbstractFetcherThread.shutdown() should not block on ReadableByteChannel.read(buffer)

2015-07-06 Thread Dong Lin (JIRA)

[ 
https://issues.apache.org/jira/browse/KAFKA-2241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14615627#comment-14615627
 ] 

Dong Lin commented on KAFKA-2241:
-

Hi [~junrao]], could you please review this patch when you have time?

> AbstractFetcherThread.shutdown() should not block on 
> ReadableByteChannel.read(buffer)
> -
>
> Key: KAFKA-2241
> URL: https://issues.apache.org/jira/browse/KAFKA-2241
> Project: Kafka
>  Issue Type: Bug
>Reporter: Dong Lin
>Assignee: Dong Lin
>  Labels: quotas
> Attachments: KAFKA-2241.patch, KAFKA-2241_2015-06-03_15:30:35.patch, 
> client.java, server.java
>
>
> This is likely a bug from Java. This affects Kafka and here is the patch to 
> fix it.
> Here is the description of the bug. By description of SocketChannel in Java 7 
> Documentation. If another thread interrupts the current thread while the read 
> operation is in progress, the it should closes the channel and throw 
> ClosedByInterruptException. However, we find that interrupting the thread 
> will not unblock the channel immediately. Instead, it waits for response or 
> socket timeout before throwing an exception.
> This will cause problem in the following scenario. Suppose one 
> console_consumer_1 is reading from a topic, and due to quota delay or 
> whatever reason, it block on channel.read(buffer). At this moment, another 
> console_consumer_2 joins and triggers rebalance at console_consumer_1. But 
> consumer_1 will block waiting on the channel.read before it can release 
> partition ownership, causing consumer_2 to fail after a number of failed 
> attempts to obtain partition ownership.
> In other words, AbstractFetcherThread.shutdown() is not guaranteed to 
> shutdown due to this bug.
> The problem is confirmed with Java 1.7 and java 1.6. To check it by yourself, 
> you can use the attached server.java and client.java -- start the server 
> before the client and see if client unblock after interruption.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Comment Edited] (KAFKA-2241) AbstractFetcherThread.shutdown() should not block on ReadableByteChannel.read(buffer)

2015-07-06 Thread Dong Lin (JIRA)

[ 
https://issues.apache.org/jira/browse/KAFKA-2241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14615627#comment-14615627
 ] 

Dong Lin edited comment on KAFKA-2241 at 7/6/15 8:48 PM:
-

Hi [~junrao], could you please review this patch when you have time?


was (Author: lindong):
Hi [~junrao]], could you please review this patch when you have time?

> AbstractFetcherThread.shutdown() should not block on 
> ReadableByteChannel.read(buffer)
> -
>
> Key: KAFKA-2241
> URL: https://issues.apache.org/jira/browse/KAFKA-2241
> Project: Kafka
>  Issue Type: Bug
>Reporter: Dong Lin
>Assignee: Dong Lin
>  Labels: quotas
> Attachments: KAFKA-2241.patch, KAFKA-2241_2015-06-03_15:30:35.patch, 
> client.java, server.java
>
>
> This is likely a bug from Java. This affects Kafka and here is the patch to 
> fix it.
> Here is the description of the bug. By description of SocketChannel in Java 7 
> Documentation. If another thread interrupts the current thread while the read 
> operation is in progress, the it should closes the channel and throw 
> ClosedByInterruptException. However, we find that interrupting the thread 
> will not unblock the channel immediately. Instead, it waits for response or 
> socket timeout before throwing an exception.
> This will cause problem in the following scenario. Suppose one 
> console_consumer_1 is reading from a topic, and due to quota delay or 
> whatever reason, it block on channel.read(buffer). At this moment, another 
> console_consumer_2 joins and triggers rebalance at console_consumer_1. But 
> consumer_1 will block waiting on the channel.read before it can release 
> partition ownership, causing consumer_2 to fail after a number of failed 
> attempts to obtain partition ownership.
> In other words, AbstractFetcherThread.shutdown() is not guaranteed to 
> shutdown due to this bug.
> The problem is confirmed with Java 1.7 and java 1.6. To check it by yourself, 
> you can use the attached server.java and client.java -- start the server 
> before the client and see if client unblock after interruption.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


Re: Review Request 36034: Patch for KAFKA-2306

2015-07-06 Thread Dong Lin


> On July 3, 2015, 1:36 a.m., Guozhang Wang wrote:
> > Thanks for the patch. I have a few thoughts regarding the names of the 
> > metrics, since in the producer other causes can also result in dropped 
> > messages (i.e. rejected before it enteres the producer buffer), such as 
> > message-size-too-large, serialization-failed, etc. In the old producer 
> > since we only have one cause we named that to droppedMessageRate. So I 
> > think we could either:
> > 
> > 1. record dropped messages for any KafkaExceptions, but not limited to 
> > BufferExhaustedException.
> > 2. have a separate metric for buffer-exhausted with a different name.
> > 
> > I prefer the first option since I feel in practice people just want to 
> > distinguish between the case that messages failed to get into the producer 
> > from the case the messages gets failed to send to the broker.
> 
> Dong Lin wrote:
> Thanks for your comments. If I understand it right, we already have a 
> sensor named "errors" which records the message dropped for any 
> KafkaExceptions. Therefore no work is needed option 1.
> 
> I think there is probably need in opertaion to track the number of 
> messages dropped due to BufferExhaustedException. Because 1) this is a common 
> exception may happen un-noticed if block_on_buffer_full=false and 
> asynchronous producer is used; and 2) for backward compatibility with old 
> producer. I will ask Joel if he has more detail on the motivation.
> 
> Guozhang Wang wrote:
> There are two cenarios that a message does not successfully reach the 
> broker:
> 
> 1. The message gets rejected by the producer immediately before being 
> added to the producer's buffer for sending. This error is thrown as 
> non-ApiException KafkaException.
> 2. The message was sent to the broker from the producer's send buffer but 
> got rejected (and later retries exhausted). This error is thrown as 
> ApiException.
> 
> Currently both of them are recorded as record-error-rate, while in the 
> old producer, we record DroppedRecord for the first scenario, which only 
> includes BufferFullException.
> 
> So I was proposing if we want back-ward compatibility we could record the 
> first scenario, which include BufferExhausted but also some other exceptions 
> in a separate metric. Does that sound reasonable?
> 
> Dong Lin wrote:
> Yeah I understand your suggestion. But could you please explain why 
> having a sensor for all non-ApiException KafkaException is better than 
> ApiException KafkaException only?
> 
> I think it wouldn't be exactly backward compatible if you include other 
> exceptions, such as ClassCastException, in this sensor. It could cause 
> problem to users if they are depending on this sensor to measure how many 
> data are dropped in asynchronous call due to BufferFullException.
> 
> What do you think?

oops. I mean SerializationException, not ClassCastException, in the comment 
above.

BTW, the addition of this sensor is motivated when we prepare the release note 
of new producer. I think if this sensor is not important, we don't need to add 
it. Otherwise, it is easier to explain to user with its original definition.


- Dong


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36034/#review90303
---


On July 3, 2015, 1:56 a.m., Dong Lin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/36034/
> ---
> 
> (Updated July 3, 2015, 1:56 a.m.)
> 
> 
> Review request for kafka and Joel Koshy.
> 
> 
> Bugs: KAFKA-2306
> https://issues.apache.org/jira/browse/KAFKA-2306
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> KAFKA-2306; New producer should emit metrics for buffer exhaustion
> 
> 
> Diffs
> -
> 
>   clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 
> 5671a3fbeea8cb9a9ffeeb41aa1b132b92c0cae8 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java 
> 0baf16e55046a2f49f6431e01d52c323c95eddf0 
> 
> Diff: https://reviews.apache.org/r/36034/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Dong Lin
> 
>



Re: Review Request 36030: Patch for KAFKA-972

2015-07-06 Thread Jun Rao

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36030/#review90579
---


Thanks for the patch. A few more minor comments blow.


core/src/test/scala/unit/kafka/integration/TopicMetadataTest.scala (line 139)


This can be private, right?



core/src/test/scala/unit/kafka/integration/TopicMetadataTest.scala (lines 140 - 
148)


This seems redundant given the code in 155 to 163. We can probaby just 
assert the broker size on topicMetadata after line 152.



core/src/test/scala/unit/kafka/integration/TopicMetadataTest.scala (line 155)


It's better to use foreach() instead map() in this case since we don't need 
the output of map.


- Jun Rao


On July 1, 2015, 3:06 p.m., Ashish Singh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/36030/
> ---
> 
> (Updated July 1, 2015, 3:06 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-972
> https://issues.apache.org/jira/browse/KAFKA-972
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> KAFKA-972: MetadataRequest returns stale list of brokers
> 
> 
> Diffs
> -
> 
>   core/src/main/scala/kafka/controller/KafkaController.scala 
> 36350579b16027359d237b64699003358704ac6f 
>   core/src/test/scala/unit/kafka/integration/TopicMetadataTest.scala 
> 995b05901491bb0dbf0df210d44bd1d7f66fdc82 
> 
> Diff: https://reviews.apache.org/r/36030/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Ashish Singh
> 
>



Re: Review Request 36034: Patch for KAFKA-2306

2015-07-06 Thread Guozhang Wang


> On July 3, 2015, 1:36 a.m., Guozhang Wang wrote:
> > Thanks for the patch. I have a few thoughts regarding the names of the 
> > metrics, since in the producer other causes can also result in dropped 
> > messages (i.e. rejected before it enteres the producer buffer), such as 
> > message-size-too-large, serialization-failed, etc. In the old producer 
> > since we only have one cause we named that to droppedMessageRate. So I 
> > think we could either:
> > 
> > 1. record dropped messages for any KafkaExceptions, but not limited to 
> > BufferExhaustedException.
> > 2. have a separate metric for buffer-exhausted with a different name.
> > 
> > I prefer the first option since I feel in practice people just want to 
> > distinguish between the case that messages failed to get into the producer 
> > from the case the messages gets failed to send to the broker.
> 
> Dong Lin wrote:
> Thanks for your comments. If I understand it right, we already have a 
> sensor named "errors" which records the message dropped for any 
> KafkaExceptions. Therefore no work is needed option 1.
> 
> I think there is probably need in opertaion to track the number of 
> messages dropped due to BufferExhaustedException. Because 1) this is a common 
> exception may happen un-noticed if block_on_buffer_full=false and 
> asynchronous producer is used; and 2) for backward compatibility with old 
> producer. I will ask Joel if he has more detail on the motivation.
> 
> Guozhang Wang wrote:
> There are two cenarios that a message does not successfully reach the 
> broker:
> 
> 1. The message gets rejected by the producer immediately before being 
> added to the producer's buffer for sending. This error is thrown as 
> non-ApiException KafkaException.
> 2. The message was sent to the broker from the producer's send buffer but 
> got rejected (and later retries exhausted). This error is thrown as 
> ApiException.
> 
> Currently both of them are recorded as record-error-rate, while in the 
> old producer, we record DroppedRecord for the first scenario, which only 
> includes BufferFullException.
> 
> So I was proposing if we want back-ward compatibility we could record the 
> first scenario, which include BufferExhausted but also some other exceptions 
> in a separate metric. Does that sound reasonable?
> 
> Dong Lin wrote:
> Yeah I understand your suggestion. But could you please explain why 
> having a sensor for all non-ApiException KafkaException is better than 
> ApiException KafkaException only?
> 
> I think it wouldn't be exactly backward compatible if you include other 
> exceptions, such as ClassCastException, in this sensor. It could cause 
> problem to users if they are depending on this sensor to measure how many 
> data are dropped in asynchronous call due to BufferFullException.
> 
> What do you think?
> 
> Dong Lin wrote:
> oops. I mean SerializationException, not ClassCastException, in the 
> comment above.
> 
> BTW, the addition of this sensor is motivated when we prepare the release 
> note of new producer. I think if this sensor is not important, we don't need 
> to add it. Otherwise, it is easier to explain to user with its original 
> definition.

I think you meant to say "why having a sensor for all non-ApiException 
KafkaException is better than BufferExhaustedException only", right?

I do not preferring to having a metric for all non-ApiExceptions, as I said in 
the first comment, we could do this (as option 1) or use a different name as 
"record-dropped-rate" since it is too general as referring to any 
non-ApiException that cause it to drop messages, for example 
"record-dropped-due-to-memory-exhausted-rate" for BufferExhaustedException only 
if users just want that (as option 2).


- Guozhang


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36034/#review90303
---


On July 3, 2015, 1:56 a.m., Dong Lin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/36034/
> ---
> 
> (Updated July 3, 2015, 1:56 a.m.)
> 
> 
> Review request for kafka and Joel Koshy.
> 
> 
> Bugs: KAFKA-2306
> https://issues.apache.org/jira/browse/KAFKA-2306
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> KAFKA-2306; New producer should emit metrics for buffer exhaustion
> 
> 
> Diffs
> -
> 
>   clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 
> 5671a3fbeea8cb9a9ffeeb41aa1b132b92c0cae8 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java 
> 0baf16e55046a2f49f6431e01d52c323c95eddf0 
> 
> Diff: https://reviews.apache.org/r/36034/diff/
> 
> 
> Testing
> ---
> 
> 
> Tha

[jira] [Updated] (KAFKA-972) MetadataRequest returns stale list of brokers

2015-07-06 Thread Jun Rao (JIRA)

 [ 
https://issues.apache.org/jira/browse/KAFKA-972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jun Rao updated KAFKA-972:
--
Status: In Progress  (was: Patch Available)

> MetadataRequest returns stale list of brokers
> -
>
> Key: KAFKA-972
> URL: https://issues.apache.org/jira/browse/KAFKA-972
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.8.0
>Reporter: Vinicius Carvalho
>Assignee: Ashish K Singh
> Attachments: BrokerMetadataTest.scala, KAFKA-972.patch, 
> KAFKA-972_2015-06-30_18:42:13.patch, KAFKA-972_2015-07-01_01:36:56.patch, 
> KAFKA-972_2015-07-01_01:42:42.patch, KAFKA-972_2015-07-01_08:06:03.patch
>
>
> When we issue an metadatarequest towards the cluster, the list of brokers is 
> stale. I mean, even when a broker is down, it's returned back to the client. 
> The following are examples of two invocations one with both brokers online 
> and the second with a broker down:
> {
> "brokers": [
> {
> "nodeId": 0,
> "host": "10.139.245.106",
> "port": 9092,
> "byteLength": 24
> },
> {
> "nodeId": 1,
> "host": "localhost",
> "port": 9093,
> "byteLength": 19
> }
> ],
> "topicMetadata": [
> {
> "topicErrorCode": 0,
> "topicName": "foozbar",
> "partitions": [
> {
> "replicas": [
> 0
> ],
> "isr": [
> 0
> ],
> "partitionErrorCode": 0,
> "partitionId": 0,
> "leader": 0,
> "byteLength": 26
> },
> {
> "replicas": [
> 1
> ],
> "isr": [
> 1
> ],
> "partitionErrorCode": 0,
> "partitionId": 1,
> "leader": 1,
> "byteLength": 26
> },
> {
> "replicas": [
> 0
> ],
> "isr": [
> 0
> ],
> "partitionErrorCode": 0,
> "partitionId": 2,
> "leader": 0,
> "byteLength": 26
> },
> {
> "replicas": [
> 1
> ],
> "isr": [
> 1
> ],
> "partitionErrorCode": 0,
> "partitionId": 3,
> "leader": 1,
> "byteLength": 26
> },
> {
> "replicas": [
> 0
> ],
> "isr": [
> 0
> ],
> "partitionErrorCode": 0,
> "partitionId": 4,
> "leader": 0,
> "byteLength": 26
> }
> ],
> "byteLength": 145
> }
> ],
> "responseSize": 200,
> "correlationId": -1000
> }
> {
> "brokers": [
> {
> "nodeId": 0,
> "host": "10.139.245.106",
> "port": 9092,
> "byteLength": 24
> },
> {
> "nodeId": 1,
> "host": "localhost",
> "port": 9093,
> "byteLength": 19
> }
> ],
> "topicMetadata": [
> {
> "topicErrorCode": 0,
> "topicName": "foozbar",
> "partitions": [
> {
> "replicas": [
> 0
> ],
> "isr": [],
> "partitionErrorCode": 5,
> "partitionId": 0,
> "leader": -1,
> "byteLength": 22
> },
> {
> "replicas": [
> 1
> ],
> "isr": [
> 1
> ],
> "partitionErrorCode": 0,
> "partitionId": 1,
> "leader": 1,
> "byteLength": 26
> },
> {
> "replicas": [
> 0
> ],
> "isr": [],
> "partitionErrorCode": 5,
> "partitionId": 2,
>   

Re: Review Request 36034: Patch for KAFKA-2306

2015-07-06 Thread Dong Lin


> On July 3, 2015, 1:36 a.m., Guozhang Wang wrote:
> > Thanks for the patch. I have a few thoughts regarding the names of the 
> > metrics, since in the producer other causes can also result in dropped 
> > messages (i.e. rejected before it enteres the producer buffer), such as 
> > message-size-too-large, serialization-failed, etc. In the old producer 
> > since we only have one cause we named that to droppedMessageRate. So I 
> > think we could either:
> > 
> > 1. record dropped messages for any KafkaExceptions, but not limited to 
> > BufferExhaustedException.
> > 2. have a separate metric for buffer-exhausted with a different name.
> > 
> > I prefer the first option since I feel in practice people just want to 
> > distinguish between the case that messages failed to get into the producer 
> > from the case the messages gets failed to send to the broker.
> 
> Dong Lin wrote:
> Thanks for your comments. If I understand it right, we already have a 
> sensor named "errors" which records the message dropped for any 
> KafkaExceptions. Therefore no work is needed option 1.
> 
> I think there is probably need in opertaion to track the number of 
> messages dropped due to BufferExhaustedException. Because 1) this is a common 
> exception may happen un-noticed if block_on_buffer_full=false and 
> asynchronous producer is used; and 2) for backward compatibility with old 
> producer. I will ask Joel if he has more detail on the motivation.
> 
> Guozhang Wang wrote:
> There are two cenarios that a message does not successfully reach the 
> broker:
> 
> 1. The message gets rejected by the producer immediately before being 
> added to the producer's buffer for sending. This error is thrown as 
> non-ApiException KafkaException.
> 2. The message was sent to the broker from the producer's send buffer but 
> got rejected (and later retries exhausted). This error is thrown as 
> ApiException.
> 
> Currently both of them are recorded as record-error-rate, while in the 
> old producer, we record DroppedRecord for the first scenario, which only 
> includes BufferFullException.
> 
> So I was proposing if we want back-ward compatibility we could record the 
> first scenario, which include BufferExhausted but also some other exceptions 
> in a separate metric. Does that sound reasonable?

Yeah I understand your suggestion. But could you please explain why having a 
sensor for all non-ApiException KafkaException is better than ApiException 
KafkaException only?

I think it wouldn't be exactly backward compatible if you include other 
exceptions, such as ClassCastException, in this sensor. It could cause problem 
to users if they are depending on this sensor to measure how many data are 
dropped in asynchronous call due to BufferFullException.

What do you think?


- Dong


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36034/#review90303
---


On July 3, 2015, 1:56 a.m., Dong Lin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/36034/
> ---
> 
> (Updated July 3, 2015, 1:56 a.m.)
> 
> 
> Review request for kafka and Joel Koshy.
> 
> 
> Bugs: KAFKA-2306
> https://issues.apache.org/jira/browse/KAFKA-2306
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> KAFKA-2306; New producer should emit metrics for buffer exhaustion
> 
> 
> Diffs
> -
> 
>   clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 
> 5671a3fbeea8cb9a9ffeeb41aa1b132b92c0cae8 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java 
> 0baf16e55046a2f49f6431e01d52c323c95eddf0 
> 
> Diff: https://reviews.apache.org/r/36034/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Dong Lin
> 
>



Re: Review Request 36034: Patch for KAFKA-2306

2015-07-06 Thread Dong Lin

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36034/
---

(Updated July 6, 2015, 9:54 p.m.)


Review request for kafka and Joel Koshy.


Bugs: KAFKA-2306
https://issues.apache.org/jira/browse/KAFKA-2306


Repository: kafka


Description
---

KAFKA-2306; New producer should emit metrics for buffer exhaustion


Diffs (updated)
-

  clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 
5671a3fbeea8cb9a9ffeeb41aa1b132b92c0cae8 
  clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java 
0baf16e55046a2f49f6431e01d52c323c95eddf0 

Diff: https://reviews.apache.org/r/36034/diff/


Testing
---


Thanks,

Dong Lin



[jira] [Updated] (KAFKA-2306) New producer should emit metrics for buffer exhaustion

2015-07-06 Thread Dong Lin (JIRA)

 [ 
https://issues.apache.org/jira/browse/KAFKA-2306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Dong Lin updated KAFKA-2306:

Attachment: KAFKA-2306_2015-07-06_14:54:01.patch

> New producer should emit metrics for buffer exhaustion
> --
>
> Key: KAFKA-2306
> URL: https://issues.apache.org/jira/browse/KAFKA-2306
> Project: Kafka
>  Issue Type: Bug
>Reporter: Dong Lin
>Assignee: Dong Lin
> Fix For: 0.8.3
>
> Attachments: KAFKA-2306.patch, KAFKA-2306_2015-07-06_14:54:01.patch
>
>
> In the old producer we have droppedMessageRate that allows user to monitor 
> the number of messages dropped when buffer is full and block on buffer full 
> is set to false. This metric is useful in operation. However, in the new 
> producer we don't have this a metric.
> The "errors" sensor in new-producers measures per-record error that is not 
> limited to those caused by BufferExhaustedException. Thus it is not good 
> enough.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (KAFKA-2306) New producer should emit metrics for buffer exhaustion

2015-07-06 Thread Dong Lin (JIRA)

[ 
https://issues.apache.org/jira/browse/KAFKA-2306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14615736#comment-14615736
 ] 

Dong Lin commented on KAFKA-2306:
-

Updated reviewboard https://reviews.apache.org/r/36034/diff/
 against branch origin/trunk

> New producer should emit metrics for buffer exhaustion
> --
>
> Key: KAFKA-2306
> URL: https://issues.apache.org/jira/browse/KAFKA-2306
> Project: Kafka
>  Issue Type: Bug
>Reporter: Dong Lin
>Assignee: Dong Lin
> Fix For: 0.8.3
>
> Attachments: KAFKA-2306.patch, KAFKA-2306_2015-07-06_14:54:01.patch
>
>
> In the old producer we have droppedMessageRate that allows user to monitor 
> the number of messages dropped when buffer is full and block on buffer full 
> is set to false. This metric is useful in operation. However, in the new 
> producer we don't have this a metric.
> The "errors" sensor in new-producers measures per-record error that is not 
> limited to those caused by BufferExhaustedException. Thus it is not good 
> enough.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


Re: Review Request 36034: Patch for KAFKA-2306

2015-07-06 Thread Dong Lin


> On July 3, 2015, 1:36 a.m., Guozhang Wang wrote:
> > Thanks for the patch. I have a few thoughts regarding the names of the 
> > metrics, since in the producer other causes can also result in dropped 
> > messages (i.e. rejected before it enteres the producer buffer), such as 
> > message-size-too-large, serialization-failed, etc. In the old producer 
> > since we only have one cause we named that to droppedMessageRate. So I 
> > think we could either:
> > 
> > 1. record dropped messages for any KafkaExceptions, but not limited to 
> > BufferExhaustedException.
> > 2. have a separate metric for buffer-exhausted with a different name.
> > 
> > I prefer the first option since I feel in practice people just want to 
> > distinguish between the case that messages failed to get into the producer 
> > from the case the messages gets failed to send to the broker.
> 
> Dong Lin wrote:
> Thanks for your comments. If I understand it right, we already have a 
> sensor named "errors" which records the message dropped for any 
> KafkaExceptions. Therefore no work is needed option 1.
> 
> I think there is probably need in opertaion to track the number of 
> messages dropped due to BufferExhaustedException. Because 1) this is a common 
> exception may happen un-noticed if block_on_buffer_full=false and 
> asynchronous producer is used; and 2) for backward compatibility with old 
> producer. I will ask Joel if he has more detail on the motivation.
> 
> Guozhang Wang wrote:
> There are two cenarios that a message does not successfully reach the 
> broker:
> 
> 1. The message gets rejected by the producer immediately before being 
> added to the producer's buffer for sending. This error is thrown as 
> non-ApiException KafkaException.
> 2. The message was sent to the broker from the producer's send buffer but 
> got rejected (and later retries exhausted). This error is thrown as 
> ApiException.
> 
> Currently both of them are recorded as record-error-rate, while in the 
> old producer, we record DroppedRecord for the first scenario, which only 
> includes BufferFullException.
> 
> So I was proposing if we want back-ward compatibility we could record the 
> first scenario, which include BufferExhausted but also some other exceptions 
> in a separate metric. Does that sound reasonable?
> 
> Dong Lin wrote:
> Yeah I understand your suggestion. But could you please explain why 
> having a sensor for all non-ApiException KafkaException is better than 
> ApiException KafkaException only?
> 
> I think it wouldn't be exactly backward compatible if you include other 
> exceptions, such as ClassCastException, in this sensor. It could cause 
> problem to users if they are depending on this sensor to measure how many 
> data are dropped in asynchronous call due to BufferFullException.
> 
> What do you think?
> 
> Dong Lin wrote:
> oops. I mean SerializationException, not ClassCastException, in the 
> comment above.
> 
> BTW, the addition of this sensor is motivated when we prepare the release 
> note of new producer. I think if this sensor is not important, we don't need 
> to add it. Otherwise, it is easier to explain to user with its original 
> definition.
> 
> Guozhang Wang wrote:
> I think you meant to say "why having a sensor for all non-ApiException 
> KafkaException is better than BufferExhaustedException only", right?
> 
> I do not preferring to having a metric for all non-ApiExceptions, as I 
> said in the first comment, we could do this (as option 1) or use a different 
> name as "record-dropped-rate" since it is too general as referring to any 
> non-ApiException that cause it to drop messages, for example 
> "record-dropped-due-to-memory-exhausted-rate" for BufferExhaustedException 
> only if users just want that (as option 2).

Yeah, that is my typo..

Thanks for the comment. Yes the "record-dropped-rate" is confusing. Could you 
please see if the updated patch addresses the issue? I used 
record-buffer-exhausted-rate since record-dropped-due-to-memory-exhausted-rate 
is probably too long.


- Dong


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36034/#review90303
---


On July 6, 2015, 9:54 p.m., Dong Lin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/36034/
> ---
> 
> (Updated July 6, 2015, 9:54 p.m.)
> 
> 
> Review request for kafka and Joel Koshy.
> 
> 
> Bugs: KAFKA-2306
> https://issues.apache.org/jira/browse/KAFKA-2306
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> KAFKA-2306; New producer should emit metrics for buffer exhaustion
> 
> 
> Diffs
> -
> 
>   clients/src/main/jav

Re: [DISCUSS] KIP-26 - Add Copycat, a connector framework for data import/export

2015-07-06 Thread Guozhang Wang
Hi Ewen,

I read through the KIP page and here are some comments on the design
section:

1. "... and Copycat does not require that all partitions be enumerated".
Not very clear about this, do you mean Copycat allows non-enumerable stream
partitions?

2. "... translates the data to Copycat's format, decides the destination
topic (and possibly partition) in Kafka." Just to confirm it seems
indicating two destination scenarios Copycat connectors should be able to
support:

a. Specific destination topics per task (e.g. as illustrated in the digram,
task 1 to topics A and B, task 2 to topics B and C).
b. Specific destination topic-partitions per task (as said in "possibly
partition", like task 1 to topicA-partition1 and topicB-partition1, task 2
to topicA-partition2 and topicB-partition2).

I understand connector developers needs to implement the dynamic mapping
coordination from the source streams to tasks, but does the mapping from
tasks to destination topic-partitions (for sinking Copycat I assume it
would be stream-partitions) also need to be implemented dynamically since
the destination stream could also change?

3. "Delivery Guarantees": depending on how we define the guarantees, it may
not only depends on the output system but also the input system. For
example, duplicates may be generated from the input systems as well. Do we
also need to consider these scenarios?

4. "Integration with Process Management": for "Resource constrained
connectors", I am not sure how it is different in deployment from
"Copycat-as-a-service"? I feel there are generally three different types:

  1) run-as-a-service: on a shared cluster equipped with some resource
manager, a Copycat framework is ever-running and users submit their
connector jobs via REST.
  2) standalone: on a single machine, start a Copycat instance with the
configured master + #.workers processes via some cmdline tool.
  3) embedded library: the Copycat code will be running on whatever the
embedding application is running on.

5. Some terminology suggestions, how about the following descriptions (no
technical difference except the CLI APIs, just some naming changes) of
Copycat:

a. Copycat developers needs to implement the "*connector*" module, which
include the "*master*" and "*worker*" logic:

  1) "master" is responsible for coordinating the assignment from the
resource stream partitions to the workers (and possibly also the assignment
from the workers to the destination stream partitions?) *dynamically*, and
  2) "worker" is responsible for polling from the assigned resource stream
partitions and pushing to the assigned destination stream partitions.

b. Copycat framework includes:

  1) The interface for the connector workers polling-from-resource and
pushing-to-destination function calls,
  2) The interface for resource management integration: it leverages the
underlying resource managers like YARN / Mesos to get a list of allocated "
*containers*".
  3) A "*connector manager*" responsible for coordinating the assignment
from the connector master / worker processes to the allocated containers
*dynamically*.

c. Copycat users need to specify the *connector configurations* through
config files or ZK / other storage systems, including #.tasks, starting
offsets, etc, and start the *connector job* with its configurations (each
job as its own configs) via the above mentioned three different modes:

  1) submit the job via REST to a Copycat service running on a shared
cluster with resource manager, or
  2) start the job in standalone mode in a single machine, with all the
master / workers running on that single machine.
  3) start a copycat instance first in embedded mode and then add
connectors, all the added connectors (i.e. their master / workers) run on
the single machine where the embedding app code is running.

d. As for the CLI APIs, we will only need one for the standalone mode since
the run-as-a-service mode will always have some resource manager to
allocate the containers.

Guozhang


On Mon, Jun 29, 2015 at 9:50 AM, Ewen Cheslack-Postava 
wrote:

> Seems like discussion has mostly quieted down on this. Any more questions,
> comments, or discussion? If nobody brings up any other issues, I'll start a
> vote thread in a day or two.
>
> -Ewen
>
> On Thu, Jun 25, 2015 at 3:36 PM, Jay Kreps  wrote:
>
> > We were talking on the call about a logo...so here I present "The
> Original
> > Copycat":
> > http://shirtoid.com/67790/the-original-copycat/
> >
> > -Jay
> >
> > On Tue, Jun 23, 2015 at 6:28 PM, Gwen Shapira 
> > wrote:
> >
> > > One more reason to have CopyCat as a separate project is to sidestep
> > > the entire "Why CopyCat and not X" discussion :)
> > >
> > > On Tue, Jun 23, 2015 at 6:26 PM, Gwen Shapira 
> > > wrote:
> > > > Re: Flume vs. CopyCat
> > > >
> > > > I would love to have an automagically-parallelizing, schema-aware
> > > > version of Flume with great reliability guarantees. Flume has good
> > > > core architecture and I'm sure that if t

[jira] [Commented] (KAFKA-2174) Wrong TopicMetadata deserialization

2015-07-06 Thread Osama Abu-Obeid (JIRA)

[ 
https://issues.apache.org/jira/browse/KAFKA-2174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14615788#comment-14615788
 ] 

Osama Abu-Obeid commented on KAFKA-2174:


Confirmed getting the same issue during consumer startup:
{noformat}
java.lang.ArrayIndexOutOfBoundsException: 26
at 
kafka.api.TopicMetadata$$anonfun$readFrom$1.apply$mcVI$sp(TopicMetadata.scala:38)
 ~[kafka_2.10-0.8.2.1.jar:na]
at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:141) 
~[scala-library-2.10.4.jar:na]
at kafka.api.TopicMetadata$.readFrom(TopicMetadata.scala:36) 
~[kafka_2.10-0.8.2.1.jar:na]
at 
kafka.api.TopicMetadataResponse$$anonfun$3.apply(TopicMetadataResponse.scala:31)
 ~[kafka_2.10-0.8.2.1.jar:na]
at 
kafka.api.TopicMetadataResponse$$anonfun$3.apply(TopicMetadataResponse.scala:31)
 ~[kafka_2.10-0.8.2.1.jar:na]
at 
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
 ~[scala-library-2.10.4.jar:na]
at 
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
 ~[scala-library-2.10.4.jar:na]
at scala.collection.immutable.Range.foreach(Range.scala:141) 
~[scala-library-2.10.4.jar:na]
at 
scala.collection.TraversableLike$class.map(TraversableLike.scala:244) 
~[scala-library-2.10.4.jar:na]
at scala.collection.AbstractTraversable.map(Traversable.scala:105) 
~[scala-library-2.10.4.jar:na]
at 
kafka.api.TopicMetadataResponse$.readFrom(TopicMetadataResponse.scala:31) 
~[kafka_2.10-0.8.2.1.jar:na]
at kafka.producer.SyncProducer.send(SyncProducer.scala:114) 
~[kafka_2.10-0.8.2.1.jar:na]
at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:58) 
[kafka_2.10-0.8.2.1.jar:na]
at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:93) 
[kafka_2.10-0.8.2.1.jar:na]
at 
kafka.consumer.ConsumerFetcherManager$LeaderFinderThread.doWork(ConsumerFetcherManager.scala:66)
 [kafka_2.10-0.8.2.1.jar:na]
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:60) 
[kafka_2.10-0.8.2.1.jar:na]
{noformat}

> Wrong TopicMetadata deserialization
> ---
>
> Key: KAFKA-2174
> URL: https://issues.apache.org/jira/browse/KAFKA-2174
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.2.1
>Reporter: Alexey Ozeritskiy
> Attachments: KAFKA-2174.patch
>
>
> TopicMetadata.readFrom assumes that ByteBuffer always contains the full set 
> of partitions but it is not true. On incomplete metadata we will get 
> java.lang.ArrayIndexOutOfBoundsException:
> {code}
> java.lang.ArrayIndexOutOfBoundsException: 47
> at 
> kafka.api.TopicMetadata$$anonfun$readFrom$1.apply$mcVI$sp(TopicMetadata.scala:38)
> at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:141)
> at kafka.api.TopicMetadata$.readFrom(TopicMetadata.scala:36)
> at 
> kafka.api.TopicMetadataResponse$$anonfun$3.apply(TopicMetadataResponse.scala:31)
> at 
> kafka.api.TopicMetadataResponse$$anonfun$3.apply(TopicMetadataResponse.scala:31)
> at 
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
> at 
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
> at scala.collection.immutable.Range.foreach(Range.scala:141)
> at 
> scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
> at scala.collection.AbstractTraversable.map(Traversable.scala:105)
> at 
> kafka.api.TopicMetadataResponse$.readFrom(TopicMetadataResponse.scala:31)
> {code}
> We sometimes get this exceptions on any broker restart (kill -TERM, 
> controlled.shutdown.enable=false).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (KAFKA-2248) Use Apache Rat to enforce copyright headers

2015-07-06 Thread Jun Rao (JIRA)

 [ 
https://issues.apache.org/jira/browse/KAFKA-2248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jun Rao updated KAFKA-2248:
---
Resolution: Fixed
Status: Resolved  (was: Patch Available)

Thanks the latest patch. +1 and committed to trunk.

> Use Apache Rat to enforce copyright headers
> ---
>
> Key: KAFKA-2248
> URL: https://issues.apache.org/jira/browse/KAFKA-2248
> Project: Kafka
>  Issue Type: Bug
>  Components: build
>Reporter: Ewen Cheslack-Postava
>Assignee: Ewen Cheslack-Postava
> Fix For: 0.8.3
>
> Attachments: KAFKA-2248.patch, KAFKA-2248_2015-06-13_14:10:23.patch
>
>
> Follow up to KAFKA-2161. Use Apache Rat during builds to make sure copyright 
> headers are present so we don't forget any and don't allow any incorrect ones 
> to be checked in.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


Build failed in Jenkins: Kafka-trunk #525

2015-07-06 Thread Apache Jenkins Server
See 

Changes:

[junrao] kafka-2248; Use Apache Rat to enforce copyright headers; patched by 
Ewen Cheslack-Postava; reviewed by Gwen Shapira, Joel Joshy and Jun Rao

--
Started by an SCM change
Building remotely on jenkins-ubuntu-1404-4gb-1e7 (jenkins-cloud-4GB cloud-slave 
Ubuntu ubuntu) in workspace 
Cloning the remote Git repository
Cloning repository https://git-wip-us.apache.org/repos/asf/kafka.git
 > git init  # timeout=10
Fetching upstream changes from https://git-wip-us.apache.org/repos/asf/kafka.git
 > git --version # timeout=10
 > git fetch --tags --progress 
 > https://git-wip-us.apache.org/repos/asf/kafka.git 
 > +refs/heads/*:refs/remotes/origin/*
 > git config remote.origin.url 
 > https://git-wip-us.apache.org/repos/asf/kafka.git # timeout=10
 > git config remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # 
 > timeout=10
 > git config remote.origin.url 
 > https://git-wip-us.apache.org/repos/asf/kafka.git # timeout=10
Fetching upstream changes from https://git-wip-us.apache.org/repos/asf/kafka.git
 > git fetch --tags --progress 
 > https://git-wip-us.apache.org/repos/asf/kafka.git 
 > +refs/heads/*:refs/remotes/origin/*
 > git rev-parse refs/remotes/origin/trunk^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/trunk^{commit} # timeout=10
Checking out Revision fd612a2d50f1ee13009395f082357403c4277164 
(refs/remotes/origin/trunk)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f fd612a2d50f1ee13009395f082357403c4277164
 > git rev-list 3f8480ccfb011eb43da774737597c597f703e11b # timeout=10
Unpacking http://services.gradle.org/distributions/gradle-2.1-bin.zip to 
/jenkins/tools/hudson.plugins.gradle.GradleInstallation/Gradle_2.1 on 
jenkins-ubuntu-1404-4gb-1e7
Setting 
GRADLE_2_1_HOME=/jenkins/tools/hudson.plugins.gradle.GradleInstallation/Gradle_2.1
[Kafka-trunk] $ /bin/bash -xe /tmp/hudson403385653373413434.sh
+ /jenkins/tools/hudson.plugins.gradle.GradleInstallation/Gradle_2.1/bin/gradle
To honour the JVM settings for this build a new JVM will be forked. Please 
consider using the daemon: 
http://gradle.org/docs/2.1/userguide/gradle_daemon.html.
Download 
https://repo1.maven.org/maven2/org/ajoberstar/grgit/0.2.3/grgit-0.2.3.pom
Download 
https://repo1.maven.org/maven2/org/eclipse/jgit/org.eclipse.jgit/3.3.0.201403021825-r/org.eclipse.jgit-3.3.0.201403021825-r.pom
Download 
https://repo1.maven.org/maven2/org/eclipse/jgit/org.eclipse.jgit-parent/3.3.0.201403021825-r/org.eclipse.jgit-parent-3.3.0.201403021825-r.pom
Download 
https://repo1.maven.org/maven2/org/eclipse/jgit/org.eclipse.jgit.ui/3.3.0.201403021825-r/org.eclipse.jgit.ui-3.3.0.201403021825-r.pom
Download 
https://repo1.maven.org/maven2/com/jcraft/jsch.agentproxy.jsch/0.0.7/jsch.agentproxy.jsch-0.0.7.pom
Download 
https://repo1.maven.org/maven2/com/jcraft/jsch.agentproxy/0.0.7/jsch.agentproxy-0.0.7.pom
Download 
https://repo1.maven.org/maven2/com/jcraft/jsch.agentproxy.pageant/0.0.7/jsch.agentproxy.pageant-0.0.7.pom
Download 
https://repo1.maven.org/maven2/com/jcraft/jsch.agentproxy.sshagent/0.0.7/jsch.agentproxy.sshagent-0.0.7.pom
Download 
https://repo1.maven.org/maven2/com/jcraft/jsch.agentproxy.usocket-jna/0.0.7/jsch.agentproxy.usocket-jna-0.0.7.pom
Download 
https://repo1.maven.org/maven2/com/jcraft/jsch.agentproxy.usocket-nc/0.0.7/jsch.agentproxy.usocket-nc-0.0.7.pom
Download https://repo1.maven.org/maven2/com/jcraft/jsch/0.1.46/jsch-0.1.46.pom
Download 
https://repo1.maven.org/maven2/com/googlecode/javaewah/JavaEWAH/0.7.9/JavaEWAH-0.7.9.pom
Download 
https://repo1.maven.org/maven2/org/apache/httpcomponents/httpclient/4.1.3/httpclient-4.1.3.pom
Download 
https://repo1.maven.org/maven2/org/apache/httpcomponents/httpcomponents-client/4.1.3/httpcomponents-client-4.1.3.pom
Download 
https://repo1.maven.org/maven2/org/apache/httpcomponents/project/5/project-5.pom
Download 
https://repo1.maven.org/maven2/com/jcraft/jsch.agentproxy.core/0.0.7/jsch.agentproxy.core-0.0.7.pom
Download 
https://repo1.maven.org/maven2/org/apache/httpcomponents/httpcore/4.1.4/httpcore-4.1.4.pom
Download 
https://repo1.maven.org/maven2/org/apache/httpcomponents/httpcomponents-core/4.1.4/httpcomponents-core-4.1.4.pom
Download 
https://repo1.maven.org/maven2/org/ajoberstar/grgit/0.2.3/grgit-0.2.3.jar
Download 
https://repo1.maven.org/maven2/org/eclipse/jgit/org.eclipse.jgit/3.3.0.201403021825-r/org.eclipse.jgit-3.3.0.201403021825-r.jar
Download 
https://repo1.maven.org/maven2/org/eclipse/jgit/org.eclipse.jgit.ui/3.3.0.201403021825-r/org.eclipse.jgit.ui-3.3.0.201403021825-r.jar
Download 
https://repo1.maven.org/maven2/com/jcraft/jsch.agentproxy.jsch/0.0.7/jsch.agentproxy.jsch-0.0.7.jar
Download 
https://repo1.maven.org/maven2/com/jcraft/jsch.agentproxy.pageant/0.0.7/jsch.agentproxy.pageant-0.0.7.jar
Download 
https://repo1.maven.org/maven

Re: [DISCUSS] KIP-26 - Add Copycat, a connector framework for data import/export

2015-07-06 Thread Ewen Cheslack-Postava
On Mon, Jul 6, 2015 at 11:40 AM, Guozhang Wang  wrote:

> Hi Ewen,
>
> I read through the KIP page and here are some comments on the design
> section:
>
> 1. "... and Copycat does not require that all partitions be enumerated".
> Not very clear about this, do you mean Copycat allows non-enumerable stream
> partitions?
>

Maybe I should change "enumerated" to just plain "listed". The point is
that the framework shouldn't ever need to ask connectors for a complete
list of their current partitions. Requiring the connector to explicitly
list all partitions can be simplifying for the framework and connectors
(e.g. we could push the work of dividing partitions over tasks into the
framework, as we do with topic-partitions in sinks), but there are some
cases where that behavior isn't ideal (e.g. JMX metrics, where an app
restart could change the set of metrics, and can cause particularly bad
behavior during a rolling restart of a service since Copycat would end up
continuously readjusting assignments).


>
> 2. "... translates the data to Copycat's format, decides the destination
> topic (and possibly partition) in Kafka." Just to confirm it seems
> indicating two destination scenarios Copycat connectors should be able to
> support:
>
> a. Specific destination topics per task (e.g. as illustrated in the digram,
> task 1 to topics A and B, task 2 to topics B and C).
> b. Specific destination topic-partitions per task (as said in "possibly
> partition", like task 1 to topicA-partition1 and topicB-partition1, task 2
> to topicA-partition2 and topicB-partition2).
>
> I understand connector developers needs to implement the dynamic mapping
> coordination from the source streams to tasks, but does the mapping from
> tasks to destination topic-partitions (for sinking Copycat I assume it
> would be stream-partitions) also need to be implemented dynamically since
> the destination stream could also change?
>

Not sure I understand what you're getting at here. Connectors can do
arbitrary shuffling to the output (which may not matter for many
connectors, e.g. HDFS, where there's only one output). Some may not need
that (e.g. reading a database commit log, you probably want to maintain
ordering within a single topic).

But as of now, there's no need to track the tasks -> destination
topic-partitions at all. There's one or two things I can think of where you
could possibly optimize them a bit in a a couple of cases if you knew this
mapping (e.g. the flush + offset commit process), but I don't think that
info is that useful to copycat.


>
> 3. "Delivery Guarantees": depending on how we define the guarantees, it may
> not only depends on the output system but also the input system. For
> example, duplicates may be generated from the input systems as well. Do we
> also need to consider these scenarios?
>

Yes, that's correct. For source connectors, if the source system introduces
duplicates then we are not doing deduplication and if it drops data there's
nothing we can do. Same deal with the output system for sink connectors. I
guess on the sink side the expected semantics are more clear since
SinkTask.flush() makes the expectations pretty clear, but on the source
side the expectation of no duplicates/dropped data is implicit.


> 4. "Integration with Process Management": for "Resource constrained
> connectors", I am not sure how it is different in deployment from
> "Copycat-as-a-service"? I feel there are generally three different types:
>
>   1) run-as-a-service: on a shared cluster equipped with some resource
> manager, a Copycat framework is ever-running and users submit their
> connector jobs via REST.
>   2) standalone: on a single machine, start a Copycat instance with the
> configured master + #.workers processes via some cmdline tool.
>   3) embedded library: the Copycat code will be running on whatever the
> embedding application is running on.
>

The reason it's different from Copycat-as-a-service is because you can
apply resource constraints *on a single, specific copycat connector*. In
"as-a-service" mode, all the connectors and tasks are mixed up across the
workers, so if you want to set a CPU or memory constraint on one
connector's tasks, you can't do that. In order to do that with a resource
manager that works at the process level and support varying constraints
(e.g. connector A gets 1 CPU, connector B gets 10 CPU), you need to make
sure the processes you are applying limits to only contain one connector's
tasks.

Because "resource constrained connectors" only runs one connector and it's
tasks, it is functionally the same as using embedded mode, not adding any
code besides Copycat to the program, and running that under the cluster
manager.


>
> 5. Some terminology suggestions, how about the following descriptions (no
> technical difference except the CLI APIs, just some naming changes) of
> Copycat:
>
> a. Copycat developers needs to implement the "*connector*" module, which
> include the "*master*" and "*worke

[jira] [Updated] (KAFKA-2132) Move Log4J appender to a separate module

2015-07-06 Thread Jun Rao (JIRA)

 [ 
https://issues.apache.org/jira/browse/KAFKA-2132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jun Rao updated KAFKA-2132:
---
   Resolution: Fixed
 Reviewer: Jun Rao  (was: Jay Kreps)
Fix Version/s: 0.8.3
   Status: Resolved  (was: Patch Available)

Thanks for the latest patch. +1 and committed to trunk.

> Move Log4J appender to a separate module
> 
>
> Key: KAFKA-2132
> URL: https://issues.apache.org/jira/browse/KAFKA-2132
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Gwen Shapira
>Assignee: Ashish K Singh
> Fix For: 0.8.3
>
> Attachments: KAFKA-2132.patch, KAFKA-2132_2015-04-27_19:59:46.patch, 
> KAFKA-2132_2015-04-30_12:22:02.patch, KAFKA-2132_2015-04-30_15:53:17.patch, 
> KAFKA-2132_2015-06-13_21:18:59.patch, KAFKA-2132_2015-06-24_10:19:56.patch, 
> KAFKA-2132_2015-06-24_10:25:43.patch, KAFKA-2132_2015-06-30_12:07:23.patch
>
>
> Log4j appender is just a producer.
> Since we have a new producer in the clients module, no need to keep Log4J 
> appender in "core" and force people to package all of Kafka with their apps.
> Lets move the Log4jAppender to clients module.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


Build failed in Jenkins: KafkaPreCommit #136

2015-07-06 Thread Apache Jenkins Server
See 

Changes:

[junrao] kafka-2248; Use Apache Rat to enforce copyright headers; patched by 
Ewen Cheslack-Postava; reviewed by Gwen Shapira, Joel Joshy and Jun Rao

[junrao] kafka-2132; Move Log4J appender to a separate module; patched by 
Ashish Singh; reviewed by Gwen Shapira, Aditya Auradkar and Jun Rao

--
Started by an SCM change
Building remotely on ubuntu-2 (docker Ubuntu ubuntu) in workspace 

Cloning the remote Git repository
Cloning repository https://git-wip-us.apache.org/repos/asf/kafka.git
 > git init  # timeout=10
Fetching upstream changes from https://git-wip-us.apache.org/repos/asf/kafka.git
 > git --version # timeout=10
 > git fetch --tags --progress 
 > https://git-wip-us.apache.org/repos/asf/kafka.git 
 > +refs/heads/*:refs/remotes/origin/*
 > git config remote.origin.url 
 > https://git-wip-us.apache.org/repos/asf/kafka.git # timeout=10
 > git config remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # 
 > timeout=10
 > git config remote.origin.url 
 > https://git-wip-us.apache.org/repos/asf/kafka.git # timeout=10
Fetching upstream changes from https://git-wip-us.apache.org/repos/asf/kafka.git
 > git fetch --tags --progress 
 > https://git-wip-us.apache.org/repos/asf/kafka.git 
 > +refs/heads/*:refs/remotes/origin/*
 > git rev-parse refs/remotes/origin/trunk^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/trunk^{commit} # timeout=10
Checking out Revision 2d96da05a0af7847aca5edc6d003a18be7f5216a 
(refs/remotes/origin/trunk)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 2d96da05a0af7847aca5edc6d003a18be7f5216a
 > git rev-list 3f8480ccfb011eb43da774737597c597f703e11b # timeout=10
Setting 
GRADLE_2_1_HOME=/home/jenkins/jenkins-slave/tools/hudson.plugins.gradle.GradleInstallation/Gradle_2.1
[KafkaPreCommit] $ /bin/bash -xe /tmp/hudson631997471208534764.sh
+ 
/home/jenkins/jenkins-slave/tools/hudson.plugins.gradle.GradleInstallation/Gradle_2.1/bin/gradle
To honour the JVM settings for this build a new JVM will be forked. Please 
consider using the daemon: 
http://gradle.org/docs/2.1/userguide/gradle_daemon.html.
Download 
https://repo1.maven.org/maven2/org/ajoberstar/grgit/0.2.3/grgit-0.2.3.pom
Download 
https://repo1.maven.org/maven2/org/eclipse/jgit/org.eclipse.jgit/3.3.0.201403021825-r/org.eclipse.jgit-3.3.0.201403021825-r.pom
Download 
https://repo1.maven.org/maven2/org/eclipse/jgit/org.eclipse.jgit-parent/3.3.0.201403021825-r/org.eclipse.jgit-parent-3.3.0.201403021825-r.pom
Download 
https://repo1.maven.org/maven2/org/eclipse/jgit/org.eclipse.jgit.ui/3.3.0.201403021825-r/org.eclipse.jgit.ui-3.3.0.201403021825-r.pom
Download 
https://repo1.maven.org/maven2/com/jcraft/jsch.agentproxy.jsch/0.0.7/jsch.agentproxy.jsch-0.0.7.pom
Download 
https://repo1.maven.org/maven2/com/jcraft/jsch.agentproxy/0.0.7/jsch.agentproxy-0.0.7.pom
Download 
https://repo1.maven.org/maven2/com/jcraft/jsch.agentproxy.pageant/0.0.7/jsch.agentproxy.pageant-0.0.7.pom
Download 
https://repo1.maven.org/maven2/com/jcraft/jsch.agentproxy.sshagent/0.0.7/jsch.agentproxy.sshagent-0.0.7.pom
Download 
https://repo1.maven.org/maven2/com/jcraft/jsch.agentproxy.usocket-jna/0.0.7/jsch.agentproxy.usocket-jna-0.0.7.pom
Download 
https://repo1.maven.org/maven2/com/jcraft/jsch.agentproxy.usocket-nc/0.0.7/jsch.agentproxy.usocket-nc-0.0.7.pom
Download https://repo1.maven.org/maven2/com/jcraft/jsch/0.1.46/jsch-0.1.46.pom
Download 
https://repo1.maven.org/maven2/com/jcraft/jsch.agentproxy.core/0.0.7/jsch.agentproxy.core-0.0.7.pom
Download 
https://repo1.maven.org/maven2/org/ajoberstar/grgit/0.2.3/grgit-0.2.3.jar
Download 
https://repo1.maven.org/maven2/org/eclipse/jgit/org.eclipse.jgit/3.3.0.201403021825-r/org.eclipse.jgit-3.3.0.201403021825-r.jar
Download 
https://repo1.maven.org/maven2/org/eclipse/jgit/org.eclipse.jgit.ui/3.3.0.201403021825-r/org.eclipse.jgit.ui-3.3.0.201403021825-r.jar
Download 
https://repo1.maven.org/maven2/com/jcraft/jsch.agentproxy.jsch/0.0.7/jsch.agentproxy.jsch-0.0.7.jar
Download 
https://repo1.maven.org/maven2/com/jcraft/jsch.agentproxy.pageant/0.0.7/jsch.agentproxy.pageant-0.0.7.jar
Download 
https://repo1.maven.org/maven2/com/jcraft/jsch.agentproxy.sshagent/0.0.7/jsch.agentproxy.sshagent-0.0.7.jar
Download 
https://repo1.maven.org/maven2/com/jcraft/jsch.agentproxy.usocket-jna/0.0.7/jsch.agentproxy.usocket-jna-0.0.7.jar
Download 
https://repo1.maven.org/maven2/com/jcraft/jsch.agentproxy.usocket-nc/0.0.7/jsch.agentproxy.usocket-nc-0.0.7.jar
Download 
https://repo1.maven.org/maven2/com/jcraft/jsch.agentproxy.core/0.0.7/jsch.agentproxy.core-0.0.7.jar
Building project 'core' with Scala version 2.10.5
:downloadWrapper

BUILD SUCCESSFUL

Total time: 25.997 secs
Setting 
GRADLE_2_1_HOME=/home/jenkins/jenkins-slave/tools/hudson.plugins.gradle.GradleInstallation/Gradle_2.1
[KafkaPreC

Re: Build failed in Jenkins: KafkaPreCommit #136

2015-07-06 Thread Jay Kreps
Ha ha, hoisted on our own petard.

-Jay

On Mon, Jul 6, 2015 at 4:39 PM, Apache Jenkins Server <
jenk...@builds.apache.org> wrote:

> See 
>
> Changes:
>
> [junrao] kafka-2248; Use Apache Rat to enforce copyright headers; patched
> by Ewen Cheslack-Postava; reviewed by Gwen Shapira, Joel Joshy and Jun Rao
>
> [junrao] kafka-2132; Move Log4J appender to a separate module; patched by
> Ashish Singh; reviewed by Gwen Shapira, Aditya Auradkar and Jun Rao
>
> --
> Started by an SCM change
> Building remotely on ubuntu-2 (docker Ubuntu ubuntu) in workspace <
> https://builds.apache.org/job/KafkaPreCommit/ws/>
> Cloning the remote Git repository
> Cloning repository https://git-wip-us.apache.org/repos/asf/kafka.git
>  > git init  #
> timeout=10
> Fetching upstream changes from
> https://git-wip-us.apache.org/repos/asf/kafka.git
>  > git --version # timeout=10
>  > git fetch --tags --progress
> https://git-wip-us.apache.org/repos/asf/kafka.git
> +refs/heads/*:refs/remotes/origin/*
>  > git config remote.origin.url
> https://git-wip-us.apache.org/repos/asf/kafka.git # timeout=10
>  > git config remote.origin.fetch +refs/heads/*:refs/remotes/origin/* #
> timeout=10
>  > git config remote.origin.url
> https://git-wip-us.apache.org/repos/asf/kafka.git # timeout=10
> Fetching upstream changes from
> https://git-wip-us.apache.org/repos/asf/kafka.git
>  > git fetch --tags --progress
> https://git-wip-us.apache.org/repos/asf/kafka.git
> +refs/heads/*:refs/remotes/origin/*
>  > git rev-parse refs/remotes/origin/trunk^{commit} # timeout=10
>  > git rev-parse refs/remotes/origin/origin/trunk^{commit} # timeout=10
> Checking out Revision 2d96da05a0af7847aca5edc6d003a18be7f5216a
> (refs/remotes/origin/trunk)
>  > git config core.sparsecheckout # timeout=10
>  > git checkout -f 2d96da05a0af7847aca5edc6d003a18be7f5216a
>  > git rev-list 3f8480ccfb011eb43da774737597c597f703e11b # timeout=10
> Setting
> GRADLE_2_1_HOME=/home/jenkins/jenkins-slave/tools/hudson.plugins.gradle.GradleInstallation/Gradle_2.1
> [KafkaPreCommit] $ /bin/bash -xe /tmp/hudson631997471208534764.sh
> +
> /home/jenkins/jenkins-slave/tools/hudson.plugins.gradle.GradleInstallation/Gradle_2.1/bin/gradle
> To honour the JVM settings for this build a new JVM will be forked. Please
> consider using the daemon:
> http://gradle.org/docs/2.1/userguide/gradle_daemon.html.
> Download
> https://repo1.maven.org/maven2/org/ajoberstar/grgit/0.2.3/grgit-0.2.3.pom
> Download
> https://repo1.maven.org/maven2/org/eclipse/jgit/org.eclipse.jgit/3.3.0.201403021825-r/org.eclipse.jgit-3.3.0.201403021825-r.pom
> Download
> https://repo1.maven.org/maven2/org/eclipse/jgit/org.eclipse.jgit-parent/3.3.0.201403021825-r/org.eclipse.jgit-parent-3.3.0.201403021825-r.pom
> Download
> https://repo1.maven.org/maven2/org/eclipse/jgit/org.eclipse.jgit.ui/3.3.0.201403021825-r/org.eclipse.jgit.ui-3.3.0.201403021825-r.pom
> Download
> https://repo1.maven.org/maven2/com/jcraft/jsch.agentproxy.jsch/0.0.7/jsch.agentproxy.jsch-0.0.7.pom
> Download
> https://repo1.maven.org/maven2/com/jcraft/jsch.agentproxy/0.0.7/jsch.agentproxy-0.0.7.pom
> Download
> https://repo1.maven.org/maven2/com/jcraft/jsch.agentproxy.pageant/0.0.7/jsch.agentproxy.pageant-0.0.7.pom
> Download
> https://repo1.maven.org/maven2/com/jcraft/jsch.agentproxy.sshagent/0.0.7/jsch.agentproxy.sshagent-0.0.7.pom
> Download
> https://repo1.maven.org/maven2/com/jcraft/jsch.agentproxy.usocket-jna/0.0.7/jsch.agentproxy.usocket-jna-0.0.7.pom
> Download
> https://repo1.maven.org/maven2/com/jcraft/jsch.agentproxy.usocket-nc/0.0.7/jsch.agentproxy.usocket-nc-0.0.7.pom
> Download
> https://repo1.maven.org/maven2/com/jcraft/jsch/0.1.46/jsch-0.1.46.pom
> Download
> https://repo1.maven.org/maven2/com/jcraft/jsch.agentproxy.core/0.0.7/jsch.agentproxy.core-0.0.7.pom
> Download
> https://repo1.maven.org/maven2/org/ajoberstar/grgit/0.2.3/grgit-0.2.3.jar
> Download
> https://repo1.maven.org/maven2/org/eclipse/jgit/org.eclipse.jgit/3.3.0.201403021825-r/org.eclipse.jgit-3.3.0.201403021825-r.jar
> Download
> https://repo1.maven.org/maven2/org/eclipse/jgit/org.eclipse.jgit.ui/3.3.0.201403021825-r/org.eclipse.jgit.ui-3.3.0.201403021825-r.jar
> Download
> https://repo1.maven.org/maven2/com/jcraft/jsch.agentproxy.jsch/0.0.7/jsch.agentproxy.jsch-0.0.7.jar
> Download
> https://repo1.maven.org/maven2/com/jcraft/jsch.agentproxy.pageant/0.0.7/jsch.agentproxy.pageant-0.0.7.jar
> Download
> https://repo1.maven.org/maven2/com/jcraft/jsch.agentproxy.sshagent/0.0.7/jsch.agentproxy.sshagent-0.0.7.jar
> Download
> https://repo1.maven.org/maven2/com/jcraft/jsch.agentproxy.usocket-jna/0.0.7/jsch.agentproxy.usocket-jna-0.0.7.jar
> Download
> https://repo1.maven.org/maven2/com/jcraft/jsch.agentproxy.usocket-nc/0.0.7/jsch.agentproxy.usocket-nc-0.0.7.jar
> Download
> https://repo1.maven.org/maven2/com/jcraft/jsch.agentproxy.core/0.0.7/jsch.

[jira] [Updated] (KAFKA-2298) Client Selector can drop connections on InvalidReceiveException without notifying NetworkClient

2015-07-06 Thread Joel Koshy (JIRA)

 [ 
https://issues.apache.org/jira/browse/KAFKA-2298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joel Koshy updated KAFKA-2298:
--
Reviewer: Joel Koshy

> Client Selector can drop connections on InvalidReceiveException without 
> notifying NetworkClient
> ---
>
> Key: KAFKA-2298
> URL: https://issues.apache.org/jira/browse/KAFKA-2298
> Project: Kafka
>  Issue Type: Bug
>Reporter: Dong Lin
>Assignee: Dong Lin
>  Labels: quotas
> Attachments: KAFKA-2298.patch, KAFKA-2298_2015-06-23_18:47:54.patch, 
> KAFKA-2298_2015-06-24_13:00:39.patch
>
>
> I run into the problem described in KAFKA-2266 when testing quota. I was told 
> the bug was fixed in KAFKA-2266 after I figured out the problem.
> But the patch provided in KAFKA-2266 probably doesn't solve all related 
> problems. From reading the code there is still one edge case where the client 
> selector can close connection in poll() without notifying NetworkClient.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (KAFKA-2310) Add config to prevent broker becoming controller

2015-07-06 Thread Joel Koshy (JIRA)

[ 
https://issues.apache.org/jira/browse/KAFKA-2310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14615922#comment-14615922
 ] 

Joel Koshy commented on KAFKA-2310:
---

Is this a dup of KAFKA-1778?  
https://issues.apache.org/jira/browse/KAFKA-1778?focusedCommentId=14565914&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14565914

> Add config to prevent broker becoming controller
> 
>
> Key: KAFKA-2310
> URL: https://issues.apache.org/jira/browse/KAFKA-2310
> Project: Kafka
>  Issue Type: Bug
>Reporter: Andrii Biletskyi
>Assignee: Andrii Biletskyi
> Attachments: KAFKA-2310.patch, KAFKA-2310_0.8.1.patch, 
> KAFKA-2310_0.8.2.patch
>
>
> The goal is to be able to specify which cluster brokers can serve as a 
> controller and which cannot. This way it will be possible to "reserve" 
> particular, not overloaded with partitions and other operations, broker as 
> controller.
> Proposed to add config _controller.eligibility_ defaulted to true (for 
> backward compatibility, since now any broker can become a controller)
> Patch will be available for trunk, 0.8.2 and 0.8.1



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


Re: Review Request 36034: Patch for KAFKA-2306

2015-07-06 Thread Guozhang Wang

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36034/#review90614
---



clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 
(line 254)


I think we do not need to create a varialbe here since it is only 
referenced once in the code. Instead we could just call the following in line 
410:

metrics.sensor("...").record();



clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java 
(lines 399 - 402)


This metric may better be added in RecordAccumulator's registerMetrics() 
rather than in Sender. Also, the naming convention is not aligned with others, 
would better be "buffer-exhausted-rate".


- Guozhang Wang


On July 6, 2015, 9:54 p.m., Dong Lin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/36034/
> ---
> 
> (Updated July 6, 2015, 9:54 p.m.)
> 
> 
> Review request for kafka and Joel Koshy.
> 
> 
> Bugs: KAFKA-2306
> https://issues.apache.org/jira/browse/KAFKA-2306
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> KAFKA-2306; New producer should emit metrics for buffer exhaustion
> 
> 
> Diffs
> -
> 
>   clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 
> 5671a3fbeea8cb9a9ffeeb41aa1b132b92c0cae8 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java 
> 0baf16e55046a2f49f6431e01d52c323c95eddf0 
> 
> Diff: https://reviews.apache.org/r/36034/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Dong Lin
> 
>



Jenkins build is back to normal : Kafka-trunk #526

2015-07-06 Thread Apache Jenkins Server
See 



Re: Review Request 35820: Patch for KAFKA-1367

2015-07-06 Thread Jun Rao

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/35820/#review90605
---


Thanks for the patch. A few more comments below.


core/src/main/scala/kafka/controller/KafkaController.scala (line 42)


Implicits can have unexpected side effects. Could we limit the scope of 
this by importing it in the context where conversion is actually used?



core/src/main/scala/kafka/utils/ReplicationUtils.scala (line 44)


It's better to make this debug level.



core/src/test/scala/unit/kafka/integration/TopicMetadataTest.scala (line 45)


This seems to be specific to testIsrAfterBrokerShutDownAndJoinsBack. Could 
we move it there?



core/src/test/scala/unit/kafka/integration/TopicMetadataTest.scala (line 138)


Can this be private?



core/src/test/scala/unit/kafka/integration/TopicMetadataTest.scala (line 147)


We should use foreach instead of map.



core/src/test/scala/unit/kafka/integration/TopicMetadataTest.scala (lines 149 - 
163)


Could we align the code a bit to make it more readable?



core/src/test/scala/unit/kafka/integration/TopicMetadataTest.scala (lines 152 - 
154)


It seems that we have to deal with the case of hostName being empty here 
too. It's probably simpler to just create BrokerEndPoint in the same way as in 
line 141.


- Jun Rao


On July 2, 2015, 12:23 a.m., Ashish Singh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/35820/
> ---
> 
> (Updated July 2, 2015, 12:23 a.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1367
> https://issues.apache.org/jira/browse/KAFKA-1367
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> KAFKA-1367: Broker topic metadata not kept in sync with ZooKeeper
> 
> 
> Diffs
> -
> 
>   core/src/main/scala/kafka/common/TopicAndPartition.scala 
> df3db912f5daef6a25b4b2dd2220d2cc3795bce6 
>   core/src/main/scala/kafka/controller/KafkaController.scala 
> 36350579b16027359d237b64699003358704ac6f 
>   core/src/main/scala/kafka/utils/ReplicationUtils.scala 
> 60687332b4c9bee4d4c0851314cfb4b02d5d3489 
>   core/src/main/scala/kafka/utils/ZkUtils.scala 
> 78475e3d5ec477cef00caeaa34ff2d196466be96 
>   core/src/test/scala/unit/kafka/integration/TopicMetadataTest.scala 
> 995b05901491bb0dbf0df210d44bd1d7f66fdc82 
>   core/src/test/scala/unit/kafka/utils/ReplicationUtilsTest.scala 
> c96c0ffd958d63c09880d436b2e5ae96f51ead36 
> 
> Diff: https://reviews.apache.org/r/35820/diff/
> 
> 
> Testing
> ---
> 
> Tested on a test cluster with 3 Kafka brokers
> 
> 
> Thanks,
> 
> Ashish Singh
> 
>



[jira] [Updated] (KAFKA-1367) Broker topic metadata not kept in sync with ZooKeeper

2015-07-06 Thread Jun Rao (JIRA)

 [ 
https://issues.apache.org/jira/browse/KAFKA-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jun Rao updated KAFKA-1367:
---
Status: In Progress  (was: Patch Available)

> Broker topic metadata not kept in sync with ZooKeeper
> -
>
> Key: KAFKA-1367
> URL: https://issues.apache.org/jira/browse/KAFKA-1367
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.1, 0.8.0
>Reporter: Ryan Berdeen
>Assignee: Ashish K Singh
>  Labels: newbie++
> Fix For: 0.8.3
>
> Attachments: KAFKA-1367.patch, KAFKA-1367.txt, 
> KAFKA-1367_2015-07-01_17:23:14.patch
>
>
> When a broker is restarted, the topic metadata responses from the brokers 
> will be incorrect (different from ZooKeeper) until a preferred replica leader 
> election.
> In the metadata, it looks like leaders are correctly removed from the ISR 
> when a broker disappears, but followers are not. Then, when a broker 
> reappears, the ISR is never updated.
> I used a variation of the Vagrant setup created by Joe Stein to reproduce 
> this with latest from the 0.8.1 branch: 
> https://github.com/also/kafka/commit/dba36a503a5e22ea039df0f9852560b4fb1e067c



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[ANNOUNCE] New Committer

2015-07-06 Thread Joe Stein
I am pleased to announce that the Apache Kafka PMC has voted to invite Gwen
Shapira as a committer and Gwen has accepted.

Please join me on welcoming and congratulating Gwen.

Thanks for the contribution both in the project (code, email, etc, etc,
etc) and in throughout the community too(other projects, conferences, etc,
etc, etc). I look forward to your continued contributions and much more to
come!

~ Joe Stein
- - - - - - - - - - - - - - - - - - -
 [image: Logo-Black.jpg]
  http://www.elodina.net
http://www.stealth.ly
- - - - - - - - - - - - - - - - - - -


Re: [ANNOUNCE] New Committer

2015-07-06 Thread Ashish Singh
Congrats Gwen!

On Monday, July 6, 2015, Joe Stein  wrote:

> I am pleased to announce that the Apache Kafka PMC has voted to invite Gwen
> Shapira as a committer and Gwen has accepted.
>
> Please join me on welcoming and congratulating Gwen.
>
> Thanks for the contribution both in the project (code, email, etc, etc,
> etc) and in throughout the community too(other projects, conferences, etc,
> etc, etc). I look forward to your continued contributions and much more to
> come!
>
> ~ Joe Stein
> - - - - - - - - - - - - - - - - - - -
>  [image: Logo-Black.jpg]
>   http://www.elodina.net
> http://www.stealth.ly
> - - - - - - - - - - - - - - - - - - -
>


-- 
Ashish 🎤h


RE: [ANNOUNCE] New Committer

2015-07-06 Thread Aditya Auradkar
Congratulations Gwen!

Aditya


From: Ashish Singh [asi...@cloudera.com]
Sent: Monday, July 06, 2015 6:16 PM
To: dev@kafka.apache.org
Subject: Re: [ANNOUNCE] New Committer

Congrats Gwen!

On Monday, July 6, 2015, Joe Stein  wrote:

> I am pleased to announce that the Apache Kafka PMC has voted to invite Gwen
> Shapira as a committer and Gwen has accepted.
>
> Please join me on welcoming and congratulating Gwen.
>
> Thanks for the contribution both in the project (code, email, etc, etc,
> etc) and in throughout the community too(other projects, conferences, etc,
> etc, etc). I look forward to your continued contributions and much more to
> come!
>
> ~ Joe Stein
> - - - - - - - - - - - - - - - - - - -
>  [image: Logo-Black.jpg]
>   http://www.elodina.net
> http://www.stealth.ly
> - - - - - - - - - - - - - - - - - - -
>


--
Ashish 🎤h


Re: [ANNOUNCE] New Committer

2015-07-06 Thread Todd Palino
Congrats, Gwen! It's definitely deserved.

-Todd


> On Jul 6, 2015, at 6:08 PM, Joe Stein  wrote:
> 
> I am pleased to announce that the Apache Kafka PMC has voted to invite Gwen
> Shapira as a committer and Gwen has accepted.
> 
> Please join me on welcoming and congratulating Gwen.
> 
> Thanks for the contribution both in the project (code, email, etc, etc,
> etc) and in throughout the community too(other projects, conferences, etc,
> etc, etc). I look forward to your continued contributions and much more to
> come!
> 
> ~ Joe Stein
> - - - - - - - - - - - - - - - - - - -
> [image: Logo-Black.jpg]
>  http://www.elodina.net
>http://www.stealth.ly
> - - - - - - - - - - - - - - - - - - -


Re: Review Request 36034: Patch for KAFKA-2306

2015-07-06 Thread Dong Lin


> On July 7, 2015, 12:25 a.m., Guozhang Wang wrote:
> > clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java, 
> > line 254
> > 
> >
> > I think we do not need to create a varialbe here since it is only 
> > referenced once in the code. Instead we could just call the following in 
> > line 410:
> > 
> > metrics.sensor("...").record();

Sure. Will fix it.


> On July 7, 2015, 12:25 a.m., Guozhang Wang wrote:
> > clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java,
> >  lines 399-402
> > 
> >
> > This metric may better be added in RecordAccumulator's 
> > registerMetrics() rather than in Sender. Also, the naming convention is not 
> > aligned with others, would better be "buffer-exhausted-rate".

Sure. I also removed per-topic BufferExhaustedException metric since 
BufferExhaustedException is irrelevant to specific topic.

Thanks!


- Dong


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36034/#review90614
---


On July 6, 2015, 9:54 p.m., Dong Lin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/36034/
> ---
> 
> (Updated July 6, 2015, 9:54 p.m.)
> 
> 
> Review request for kafka and Joel Koshy.
> 
> 
> Bugs: KAFKA-2306
> https://issues.apache.org/jira/browse/KAFKA-2306
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> KAFKA-2306; New producer should emit metrics for buffer exhaustion
> 
> 
> Diffs
> -
> 
>   clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 
> 5671a3fbeea8cb9a9ffeeb41aa1b132b92c0cae8 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java 
> 0baf16e55046a2f49f6431e01d52c323c95eddf0 
> 
> Diff: https://reviews.apache.org/r/36034/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Dong Lin
> 
>



Re: Review Request 36034: Patch for KAFKA-2306

2015-07-06 Thread Dong Lin

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36034/
---

(Updated July 7, 2015, 1:22 a.m.)


Review request for kafka and Joel Koshy.


Bugs: KAFKA-2306
https://issues.apache.org/jira/browse/KAFKA-2306


Repository: kafka


Description
---

KAFKA-2306; New producer should emit metrics for buffer exhaustion


Diffs (updated)
-

  clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 
5671a3fbeea8cb9a9ffeeb41aa1b132b92c0cae8 
  
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java
 87dbd64f30f35dbf31d3820f9819a63c6c0d1e58 

Diff: https://reviews.apache.org/r/36034/diff/


Testing
---


Thanks,

Dong Lin



[jira] [Updated] (KAFKA-2306) New producer should emit metrics for buffer exhaustion

2015-07-06 Thread Dong Lin (JIRA)

 [ 
https://issues.apache.org/jira/browse/KAFKA-2306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Dong Lin updated KAFKA-2306:

Attachment: KAFKA-2306_2015-07-06_18:21:43.patch

> New producer should emit metrics for buffer exhaustion
> --
>
> Key: KAFKA-2306
> URL: https://issues.apache.org/jira/browse/KAFKA-2306
> Project: Kafka
>  Issue Type: Bug
>Reporter: Dong Lin
>Assignee: Dong Lin
> Fix For: 0.8.3
>
> Attachments: KAFKA-2306.patch, KAFKA-2306_2015-07-06_14:54:01.patch, 
> KAFKA-2306_2015-07-06_18:21:43.patch
>
>
> In the old producer we have droppedMessageRate that allows user to monitor 
> the number of messages dropped when buffer is full and block on buffer full 
> is set to false. This metric is useful in operation. However, in the new 
> producer we don't have this a metric.
> The "errors" sensor in new-producers measures per-record error that is not 
> limited to those caused by BufferExhaustedException. Thus it is not good 
> enough.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (KAFKA-2306) New producer should emit metrics for buffer exhaustion

2015-07-06 Thread Dong Lin (JIRA)

[ 
https://issues.apache.org/jira/browse/KAFKA-2306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14616028#comment-14616028
 ] 

Dong Lin commented on KAFKA-2306:
-

Updated reviewboard https://reviews.apache.org/r/36034/diff/
 against branch origin/trunk

> New producer should emit metrics for buffer exhaustion
> --
>
> Key: KAFKA-2306
> URL: https://issues.apache.org/jira/browse/KAFKA-2306
> Project: Kafka
>  Issue Type: Bug
>Reporter: Dong Lin
>Assignee: Dong Lin
> Fix For: 0.8.3
>
> Attachments: KAFKA-2306.patch, KAFKA-2306_2015-07-06_14:54:01.patch, 
> KAFKA-2306_2015-07-06_18:21:43.patch
>
>
> In the old producer we have droppedMessageRate that allows user to monitor 
> the number of messages dropped when buffer is full and block on buffer full 
> is set to false. This metric is useful in operation. However, in the new 
> producer we don't have this a metric.
> The "errors" sensor in new-producers measures per-record error that is not 
> limited to those caused by BufferExhaustedException. Thus it is not good 
> enough.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


Re: [DISCUSS] KIP-26 - Add Copycat, a connector framework for data import/export

2015-07-06 Thread Guozhang Wang
On Mon, Jul 6, 2015 at 4:33 PM, Ewen Cheslack-Postava 
wrote:

> On Mon, Jul 6, 2015 at 11:40 AM, Guozhang Wang  wrote:
>
> > Hi Ewen,
> >
> > I read through the KIP page and here are some comments on the design
> > section:
> >
> > 1. "... and Copycat does not require that all partitions be enumerated".
> > Not very clear about this, do you mean Copycat allows non-enumerable
> stream
> > partitions?
> >
>
> Maybe I should change "enumerated" to just plain "listed". The point is
> that the framework shouldn't ever need to ask connectors for a complete
> list of their current partitions. Requiring the connector to explicitly
> list all partitions can be simplifying for the framework and connectors
> (e.g. we could push the work of dividing partitions over tasks into the
> framework, as we do with topic-partitions in sinks), but there are some
> cases where that behavior isn't ideal (e.g. JMX metrics, where an app
> restart could change the set of metrics, and can cause particularly bad
> behavior during a rolling restart of a service since Copycat would end up
> continuously readjusting assignments).
>
>
>
Makes sense.


> >
> > 2. "... translates the data to Copycat's format, decides the destination
> > topic (and possibly partition) in Kafka." Just to confirm it seems
> > indicating two destination scenarios Copycat connectors should be able to
> > support:
> >
> > a. Specific destination topics per task (e.g. as illustrated in the
> digram,
> > task 1 to topics A and B, task 2 to topics B and C).
> > b. Specific destination topic-partitions per task (as said in "possibly
> > partition", like task 1 to topicA-partition1 and topicB-partition1, task
> 2
> > to topicA-partition2 and topicB-partition2).
> >
> > I understand connector developers needs to implement the dynamic mapping
> > coordination from the source streams to tasks, but does the mapping from
> > tasks to destination topic-partitions (for sinking Copycat I assume it
> > would be stream-partitions) also need to be implemented dynamically since
> > the destination stream could also change?
> >
>
> Not sure I understand what you're getting at here. Connectors can do
> arbitrary shuffling to the output (which may not matter for many
> connectors, e.g. HDFS, where there's only one output). Some may not need
> that (e.g. reading a database commit log, you probably want to maintain
> ordering within a single topic).
>
> But as of now, there's no need to track the tasks -> destination
> topic-partitions at all. There's one or two things I can think of where you
> could possibly optimize them a bit in a a couple of cases if you knew this
> mapping (e.g. the flush + offset commit process), but I don't think that
> info is that useful to copycat.
>
>
>
>From your diagrams different tasks can push to different output streams
(for source Copycat they are just output topics) or stream-partitions, so I
was asking how this is done in practice. But from your reply it seems at
least the first version of Copycat would not support that, i.e. all tasks
will be pushing to the same stream(s), and if the streams are partitioned
all tasks will be pushing to all partitions?


> >
> > 3. "Delivery Guarantees": depending on how we define the guarantees, it
> may
> > not only depends on the output system but also the input system. For
> > example, duplicates may be generated from the input systems as well. Do
> we
> > also need to consider these scenarios?
> >
>
> Yes, that's correct. For source connectors, if the source system introduces
> duplicates then we are not doing deduplication and if it drops data there's
> nothing we can do. Same deal with the output system for sink connectors. I
> guess on the sink side the expected semantics are more clear since
> SinkTask.flush() makes the expectations pretty clear, but on the source
> side the expectation of no duplicates/dropped data is implicit.
>
>
OK.


>
> > 4. "Integration with Process Management": for "Resource constrained
> > connectors", I am not sure how it is different in deployment from
> > "Copycat-as-a-service"? I feel there are generally three different types:
> >
> >   1) run-as-a-service: on a shared cluster equipped with some resource
> > manager, a Copycat framework is ever-running and users submit their
> > connector jobs via REST.
> >   2) standalone: on a single machine, start a Copycat instance with the
> > configured master + #.workers processes via some cmdline tool.
> >   3) embedded library: the Copycat code will be running on whatever the
> > embedding application is running on.
> >
>
> The reason it's different from Copycat-as-a-service is because you can
> apply resource constraints *on a single, specific copycat connector*. In
> "as-a-service" mode, all the connectors and tasks are mixed up across the
> workers, so if you want to set a CPU or memory constraint on one
> connector's tasks, you can't do that. In order to do that with a resource
> manager that works at the process level and suppor

Re: [ANNOUNCE] New Committer

2015-07-06 Thread Neha Narkhede
Very well deserved. Go Gwen!

On Mon, Jul 6, 2015 at 6:20 PM, Todd Palino  wrote:

> Congrats, Gwen! It's definitely deserved.
>
> -Todd
>
>
> > On Jul 6, 2015, at 6:08 PM, Joe Stein  wrote:
> >
> > I am pleased to announce that the Apache Kafka PMC has voted to invite
> Gwen
> > Shapira as a committer and Gwen has accepted.
> >
> > Please join me on welcoming and congratulating Gwen.
> >
> > Thanks for the contribution both in the project (code, email, etc, etc,
> > etc) and in throughout the community too(other projects, conferences,
> etc,
> > etc, etc). I look forward to your continued contributions and much more
> to
> > come!
> >
> > ~ Joe Stein
> > - - - - - - - - - - - - - - - - - - -
> > [image: Logo-Black.jpg]
> >  http://www.elodina.net
> >http://www.stealth.ly
> > - - - - - - - - - - - - - - - - - - -
>



-- 
Thanks,
Neha


Re: [ANNOUNCE] New Committer

2015-07-06 Thread Joel Koshy
Congrats Gwen!

On Mon, Jul 06, 2015 at 06:08:11PM -0700, Joe Stein wrote:
> I am pleased to announce that the Apache Kafka PMC has voted to invite Gwen
> Shapira as a committer and Gwen has accepted.
> 
> Please join me on welcoming and congratulating Gwen.
> 
> Thanks for the contribution both in the project (code, email, etc, etc,
> etc) and in throughout the community too(other projects, conferences, etc,
> etc, etc). I look forward to your continued contributions and much more to
> come!
> 
> ~ Joe Stein
> - - - - - - - - - - - - - - - - - - -
>  [image: Logo-Black.jpg]
>   http://www.elodina.net
> http://www.stealth.ly
> - - - - - - - - - - - - - - - - - - -



Re: [DISCUSS] KIP-26 - Add Copycat, a connector framework for data import/export

2015-07-06 Thread Ewen Cheslack-Postava
On Mon, Jul 6, 2015 at 6:24 PM, Guozhang Wang  wrote:

> On Mon, Jul 6, 2015 at 4:33 PM, Ewen Cheslack-Postava 
> wrote:
>
> > On Mon, Jul 6, 2015 at 11:40 AM, Guozhang Wang 
> wrote:
> >
> > > Hi Ewen,
> > >
> > > I read through the KIP page and here are some comments on the design
> > > section:
> > >
> > > 1. "... and Copycat does not require that all partitions be
> enumerated".
> > > Not very clear about this, do you mean Copycat allows non-enumerable
> > stream
> > > partitions?
> > >
> >
> > Maybe I should change "enumerated" to just plain "listed". The point is
> > that the framework shouldn't ever need to ask connectors for a complete
> > list of their current partitions. Requiring the connector to explicitly
> > list all partitions can be simplifying for the framework and connectors
> > (e.g. we could push the work of dividing partitions over tasks into the
> > framework, as we do with topic-partitions in sinks), but there are some
> > cases where that behavior isn't ideal (e.g. JMX metrics, where an app
> > restart could change the set of metrics, and can cause particularly bad
> > behavior during a rolling restart of a service since Copycat would end up
> > continuously readjusting assignments).
> >
> >
> >
> Makes sense.
>
>
> > >
> > > 2. "... translates the data to Copycat's format, decides the
> destination
> > > topic (and possibly partition) in Kafka." Just to confirm it seems
> > > indicating two destination scenarios Copycat connectors should be able
> to
> > > support:
> > >
> > > a. Specific destination topics per task (e.g. as illustrated in the
> > digram,
> > > task 1 to topics A and B, task 2 to topics B and C).
> > > b. Specific destination topic-partitions per task (as said in "possibly
> > > partition", like task 1 to topicA-partition1 and topicB-partition1,
> task
> > 2
> > > to topicA-partition2 and topicB-partition2).
> > >
> > > I understand connector developers needs to implement the dynamic
> mapping
> > > coordination from the source streams to tasks, but does the mapping
> from
> > > tasks to destination topic-partitions (for sinking Copycat I assume it
> > > would be stream-partitions) also need to be implemented dynamically
> since
> > > the destination stream could also change?
> > >
> >
> > Not sure I understand what you're getting at here. Connectors can do
> > arbitrary shuffling to the output (which may not matter for many
> > connectors, e.g. HDFS, where there's only one output). Some may not need
> > that (e.g. reading a database commit log, you probably want to maintain
> > ordering within a single topic).
> >
> > But as of now, there's no need to track the tasks -> destination
> > topic-partitions at all. There's one or two things I can think of where
> you
> > could possibly optimize them a bit in a a couple of cases if you knew
> this
> > mapping (e.g. the flush + offset commit process), but I don't think that
> > info is that useful to copycat.
> >
> >
> >
> From your diagrams different tasks can push to different output streams
> (for source Copycat they are just output topics) or stream-partitions, so I
> was asking how this is done in practice. But from your reply it seems at
> least the first version of Copycat would not support that, i.e. all tasks
> will be pushing to the same stream(s), and if the streams are partitioned
> all tasks will be pushing to all partitions?
>

Ah, I think I see. Unlike the inputs where partitions are balanced over
workers, there's no equivalent concept for outputs. In practice, source
connectors write to different topics/partitions just as they would with a
normal Kafka producer (the actual class is different since it carries a bit
of extra info about the source partition & offset, but the relevant info
just gets converted directly to a ProducerRecord). So in general, yes, all
tasks can push to all partitions.

In practice, tasks may end up only writing to a subset because of their
assignment of partitions. The JDBC sample is a good example of this -- one
topic per table might be reasonable, in which case each task "owns" and is
the only writer to the set of topics corresponding to its tables.

In the case of sink connectors/tasks, the mapping to a partitioned stream
isn't even all that strict since writing outputs is entirely under the
control of the tasks. For something like HDFS, there are output "streams"
(which are a series of files), in which case there is strict ownership
since there can only be one writer at a time. Something like Elasticsearch
or any of the NoSQL data stores might lie at the other end of the spectrum
where there is, in a sense, only one big partition that all tasks write to
(i.e. the entire data store).

>
>
>
> > >
> > > 3. "Delivery Guarantees": depending on how we define the guarantees, it
> > may
> > > not only depends on the output system but also the input system. For
> > > example, duplicates may be generated from the input systems as well. Do
> > we
> > > also need to consider these scenar

Re: [ANNOUNCE] New Committer

2015-07-06 Thread Guozhang Wang
Congrats and welcome Gwen! Very-well deserved.

Guozhang

On Mon, Jul 6, 2015 at 6:16 PM, Ashish Singh  wrote:

> Congrats Gwen!
>
> On Monday, July 6, 2015, Joe Stein  wrote:
>
> > I am pleased to announce that the Apache Kafka PMC has voted to invite
> Gwen
> > Shapira as a committer and Gwen has accepted.
> >
> > Please join me on welcoming and congratulating Gwen.
> >
> > Thanks for the contribution both in the project (code, email, etc, etc,
> > etc) and in throughout the community too(other projects, conferences,
> etc,
> > etc, etc). I look forward to your continued contributions and much more
> to
> > come!
> >
> > ~ Joe Stein
> > - - - - - - - - - - - - - - - - - - -
> >  [image: Logo-Black.jpg]
> >   http://www.elodina.net
> > http://www.stealth.ly
> > - - - - - - - - - - - - - - - - - - -
> >
>
>
> --
> Ashish 🎤h
>



-- 
-- Guozhang


Re: [ANNOUNCE] New Committer

2015-07-06 Thread Jaikiran Pai

Congratulations Gwen.

Gwen has been very helpful in various places (blogs, user mailing lists) 
which has helped me (and I'm sure many others) in using Kafka and even 
contributing patches. Very well deserved promotion.


-Jaikiran
On Tuesday 07 July 2015 07:36 AM, Guozhang Wang wrote:

Congrats and welcome Gwen! Very-well deserved.

Guozhang

On Mon, Jul 6, 2015 at 6:16 PM, Ashish Singh  wrote:


Congrats Gwen!

On Monday, July 6, 2015, Joe Stein  wrote:


I am pleased to announce that the Apache Kafka PMC has voted to invite

Gwen

Shapira as a committer and Gwen has accepted.

Please join me on welcoming and congratulating Gwen.

Thanks for the contribution both in the project (code, email, etc, etc,
etc) and in throughout the community too(other projects, conferences,

etc,

etc, etc). I look forward to your continued contributions and much more

to

come!

~ Joe Stein
- - - - - - - - - - - - - - - - - - -
  [image: Logo-Black.jpg]
   http://www.elodina.net
 http://www.stealth.ly
- - - - - - - - - - - - - - - - - - -



--
Ashish 🎤h








Re: Review Request 34554: Patch for KAFKA-2205

2015-07-06 Thread Jun Rao

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34554/#review90622
---


Thanks for the patch. A few more comments below.

1. The patch doesn't apply. Could you rebase?
2. Also, we need the logic to read all existing client configs. Is that in a 
separate jira?


core/src/main/scala/kafka/admin/ConfigCommand.scala (line 49)


What is the 1 at the end?



core/src/main/scala/kafka/admin/ConfigCommand.scala (lines 60 - 106)


It seems that all those methods can be private.



core/src/main/scala/kafka/admin/TopicCommand.scala (line 243)


The description needs to be changed to reflect what can now be altered.



core/src/main/scala/kafka/admin/TopicCommand.scala (line 252)


Since topic config can't be altered here, we need to change the description 
accordingly.



core/src/main/scala/kafka/admin/TopicCommand.scala (line 258)


Since we can't alter the config, it seems that we don't need this option at 
all.



core/src/main/scala/kafka/server/ConfigHandler.scala (line 28)


case k: (TopicAndPartition, Log) => k._1.topic

can just be 

case (topicAndPartition, log) => topicAndPartition.topic



core/src/main/scala/kafka/server/ConfigHandler.scala (line 31)


Not needed?



core/src/main/scala/kafka/server/KafkaServer.scala (line 31)


Do we need JavaConversions? If this is needed, it would be better to import 
it in the context where the conversion is actually needed.



core/src/main/scala/kafka/server/TopicConfigManager.scala (line 133)


entityType => entity_type.

Also, it would be useful to include the original json.



core/src/main/scala/kafka/server/TopicConfigManager.scala (line 138)


Value => entity_name



core/src/main/scala/kafka/server/TopicConfigManager.scala (lines 140 - 143)


Since we have verified the entity type before, we don't need to check the 
handler exists or not here.



core/src/main/scala/kafka/server/TopicConfigManager.scala (line 145)


It's probably useful to include the orginal json string. Also, could we 
make the message string in all IllegalArgumentException consistent? For 
example, they should all reference config change notification.



core/src/test/scala/unit/kafka/server/DynamicConfigChangeTest.scala (lines 88 - 
96)


Are we really mocking zkclient calls here?


- Jun Rao


On July 2, 2015, 1:39 a.m., Aditya Auradkar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/34554/
> ---
> 
> (Updated July 2, 2015, 1:39 a.m.)
> 
> 
> Review request for kafka and Joel Koshy.
> 
> 
> Bugs: KAFKA-2205
> https://issues.apache.org/jira/browse/KAFKA-2205
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> KAFKA-2205. Summary of changes:
> 
> 1. Generalized TopicConfigManager to DynamicConfigManager. It is now able to 
> handle multiple types of entities.
> 2. Changed format of the notification znode as described in KIP-21
> 3. Replaced TopicConfigManager with DynamicConfigManager.
> 4. Added new testcases. Existing testcases all pass
> 5. Added ConfigCommand to handle all config changes. Eventually this will 
> make calls to the broker once the new API's are built for now it speaks to ZK 
> directly
> 
> 
> Diffs
> -
> 
>   core/src/main/scala/kafka/admin/AdminUtils.scala 
> f06edf41c732a7b794e496d0048b0ce6f897e72b 
>   core/src/main/scala/kafka/admin/ConfigCommand.scala PRE-CREATION 
>   core/src/main/scala/kafka/admin/TopicCommand.scala 
> 8e6f18633b25bf1beee3f813b28ef7aa7d779d7b 
>   core/src/main/scala/kafka/cluster/Partition.scala 
> 730a232482fdf77be5704cdf5941cfab3828db88 
>   core/src/main/scala/kafka/controller/KafkaController.scala 
> 69bba243a9a511cc5292b43da0cc48e421a428b0 
>   core/src/main/scala/kafka/controller/PartitionLeaderSelector.scala 
> 3b15ab4eef22c6f50a7483e99a6af40fb55aca9f 
>   core/src/main/scala/kafka/controller/TopicDeletionManager.scala 
> 64ecb499f24bc801d48f86e1612d927cc08e006d 
>   core/src/main/scala/kafka/server/ConfigHandler.scala PRE-CREATION 
>   core/src/main/scala/kafka/serve

Re: Deprecation of ConsumerOffsetChecker

2015-07-06 Thread Jaikiran Pai

Thanks for explaining, Ewen.

-Jaikiran
On Thursday 02 July 2015 02:03 PM, Ewen Cheslack-Postava wrote:

Jaikiran,

After the last KIP discussion, I've been tasked with proposing a policy to
address general rules across all public interfaces including APIs, configs,
metrics, and command line interfaces, but haven't had a chance to send out
a concrete proposal yet.

I'll send that out to the dev list soon, but changes made now need some
special consideration anyway since Kafka releases up to now haven't made
any compatibility guarantees. Semantic versioning is probably a good
baseline, but even the choice of 0.8.3 vs. 0.9.0 for our next release is
pretty arbitrary since Kafka has so far made breaking changes even in minor
releases.

Until we resolve the more general compatibility rules, I'd suggest we let
this particular issue progress with the basic approach of providing one
minor point release of compatibility -- guarantee compatibility with the
previous interface for one release, providing a window for transition, then
removing the functionality in the subsequent release. For now, this just
means adding the JIRA to 0.9.x that removes the functionality, and if we
decide on a different policy, we can just adjust the Fix Version for the
removal of the old tool.

-Ewen

On Thu, Jul 2, 2015 at 12:19 AM, Jaikiran Pai 
wrote:


Just curious about deprecation policy and the version schemes. Consider a
certain feature was deprecated in 0.9.2, so a WARN gets logged in that
version. Does that now mean a (bug fix release) 0.9.3 will drop that
feature? Shouldn't the dropping of the feature be done in a 0.10.0 release
instead?

-Jaikiran

On Thursday 02 July 2015 06:27 AM, Ashish Singh wrote:


Hey Guys,

In last KIP hangout, we decided on following path for deprecating
ConsumerOffsetChecker.

1. Add deprecation warning to the tool for one release. In this case, the
warning will be added in 0.9.0.
2. Drop it completely in next release, 0.9.1.

I have updated the (KIP-23){
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=56852556
}
accordingly. {KAFKA-2307}(
https://issues.apache.org/jira/browse/KAFKA-2307)
is to remind up that we need to drop the tool in 0.9.1.

Let me know if I am missing out on any step that we decided on for the
deprecation.








Re: [ANNOUNCE] New Committer

2015-07-06 Thread Jiangjie Qin
Congrats, Gwen!

On 7/6/15, 7:17 PM, "Jaikiran Pai"  wrote:

>Congratulations Gwen.
>
>Gwen has been very helpful in various places (blogs, user mailing lists)
>which has helped me (and I'm sure many others) in using Kafka and even
>contributing patches. Very well deserved promotion.
>
>-Jaikiran
>On Tuesday 07 July 2015 07:36 AM, Guozhang Wang wrote:
>> Congrats and welcome Gwen! Very-well deserved.
>>
>> Guozhang
>>
>> On Mon, Jul 6, 2015 at 6:16 PM, Ashish Singh 
>>wrote:
>>
>>> Congrats Gwen!
>>>
>>> On Monday, July 6, 2015, Joe Stein  wrote:
>>>
 I am pleased to announce that the Apache Kafka PMC has voted to invite
>>> Gwen
 Shapira as a committer and Gwen has accepted.

 Please join me on welcoming and congratulating Gwen.

 Thanks for the contribution both in the project (code, email, etc,
etc,
 etc) and in throughout the community too(other projects, conferences,
>>> etc,
 etc, etc). I look forward to your continued contributions and much
more
>>> to
 come!

 ~ Joe Stein
 - - - - - - - - - - - - - - - - - - -
   [image: Logo-Black.jpg]
http://www.elodina.net
  http://www.stealth.ly
 - - - - - - - - - - - - - - - - - - -

>>>
>>> --
>>> Ashish 🎤h
>>>
>>
>>
>



Re: [VOTE] KIP-23 - Add JSON/CSV output and looping options to ConsumerGroupCommand

2015-07-06 Thread Ashish Singh
Thanks for your comments and vote guys. KIP-23 passed retroactively with 2
binding +1s and 2 non-binding +1s.

On Mon, Jun 29, 2015 at 3:25 PM, Jun Rao  wrote:

> +1
>
> Thanks,
>
> Jun
>
> On Tue, Jun 23, 2015 at 11:15 AM, Ashish Singh 
> wrote:
>
>> Hey Guys,
>>
>> We had some discussion over mail and in KIP hangout. I will update the RB
>> with proposed changes.
>>
>>
>> On Sun, Jun 14, 2015 at 10:07 AM, Ashish Singh 
>> wrote:
>>
>>> Hi Neha,
>>>
>>> Answers inline.
>>>
>>> On Thu, Jun 11, 2015 at 7:20 PM, Neha Narkhede 
>>> wrote:
>>>
 Thanks for submitting the KIP, Ashish! Few questions.

 1. Can you specify more details around how you expect csv output to be
 used. Same for json.

>>> CSV takes less storage space and is more convenient for shell
>>> operations. A simple diff between two csv outputs would tell you if
>>> something changed or not. It's also common in certain industries when
>>> dealing with legacy systems and workflows. Try importing JSON into MS Excel.
>>>
>>> JSON on the other hand has easy interpretation, compact notation and
>>> supports Hierarchical Data. If someone is planning to run the tool
>>> periodically and send the output to some server or even just persist it
>>> somewhere, JSON is probably the way to go.
>>>
>>> 2. If we add these options, would you still need the old format. If
 csv/json offers more convenience, should we have a plan to phase out the
 old format?

>>> Probably not, but having it around will not hurt. Having three output
>>> formats is not that bad and I do not expect this list to grow in future.
>>>

 On Thu, Jun 11, 2015 at 6:05 PM, Ashish Singh 
 wrote:

 > Jun,
 >
 > Can we add this as part of next KIP's agenda?
 >
 > On Thu, Jun 11, 2015 at 3:00 PM, Gwen Shapira 
 > wrote:
 >
 > > Maybe bring it up at the next KIP call, to make sure everyone is
 aware?
 > >
 > > On Thu, Jun 11, 2015 at 2:17 PM, Ashish Singh 
 > wrote:
 > > > Hi Guys,
 > > >
 > > > This has been lying around for quite some time. Should I start a
 voting
 > > > thread on this?
 > > >
 > > > On Thu, May 7, 2015 at 12:20 PM, Ashish Singh <
 asi...@cloudera.com>
 > > wrote:
 > > >
 > > >> Had to change the title of the page and that surprisingly
 changed the
 > > link
 > > >> as well. KIP-23 is now available at here
 > > >> <
 > >
 >
 https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=56852556
 >
 > > >> .
 > > >>
 > > >> On Thu, May 7, 2015 at 11:34 AM, Ashish Singh <
 asi...@cloudera.com>
 > > wrote:
 > > >>
 > > >>> Hi Guys,
 > > >>>
 > > >>> I just added a KIP, KIP-23 - Add JSON/CSV output and looping
 options
 > to
 > > >>> ConsumerGroupCommand
 > > >>> , for
 > > KAFKA-313
 > > >>> . The changes
 made
 > as
 > > >>> part of the JIRA can be found here <
 > > https://reviews.apache.org/r/28096/>.
 > > >>>
 > > >>> Comments and suggestions are welcome!
 > > >>>
 > > >>> --
 > > >>>
 > > >>> Regards,
 > > >>> Ashish
 > > >>>
 > > >>
 > > >>
 > > >>
 > > >> --
 > > >>
 > > >> Regards,
 > > >> Ashish
 > > >>
 > > >
 > > >
 > > >
 > > > --
 > > >
 > > > Regards,
 > > > Ashish
 > >
 >
 >
 >
 > --
 >
 > Regards,
 > Ashish
 >



 --
 Thanks,
 Neha

>>>
>>>
>>>
>>> --
>>>
>>> Regards,
>>> Ashish
>>>
>>
>>
>>
>> --
>>
>> Regards,
>> Ashish
>>
>
>


-- 

Regards,
Ashish


[jira] [Commented] (KAFKA-2129) Consumer could make multiple concurrent metadata requests

2015-07-06 Thread Tim Brooks (JIRA)

[ 
https://issues.apache.org/jira/browse/KAFKA-2129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14616136#comment-14616136
 ] 

Tim Brooks commented on KAFKA-2129:
---

Since post-2168 it seems to be explicitly documented that the consumer is not 
thread safe, and every public method (besides wakeup) requires a thread to gain 
ownership before entering the consumer, this ticket seems to be resolved.

- Tim

> Consumer could make multiple concurrent metadata requests
> -
>
> Key: KAFKA-2129
> URL: https://issues.apache.org/jira/browse/KAFKA-2129
> Project: Kafka
>  Issue Type: Bug
>  Components: clients
>Reporter: Tim Brooks
>Assignee: Tim Brooks
> Attachments: KAFKA-2129.patch
>
>
> The NetworkClient's metadataFetchInProgress is neither volatile nor atomic. 
> This protects against multiple metadata requests being made and is read on 
> poll() on the NetworkClient. It is written to when a request is initiated.
> This is fine for the producer. Which seems to have one thread writing. The 
> KafkaConsumer's poll()  method is synchronized, so there will not be more 
> than one writer entering from there. However, the NetworkClient's poll() 
> method is also accessed on the Consumer's partitionsFor() method. Which could 
> be access by a separate thread.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Created] (KAFKA-2311) Consumer's ensureNotClosed method not thread safe

2015-07-06 Thread Tim Brooks (JIRA)
Tim Brooks created KAFKA-2311:
-

 Summary: Consumer's ensureNotClosed method not thread safe
 Key: KAFKA-2311
 URL: https://issues.apache.org/jira/browse/KAFKA-2311
 Project: Kafka
  Issue Type: Bug
  Components: clients
Reporter: Tim Brooks


When a call is to the consumer is made, the first check is to see that the 
consumer is not closed. This variable is not volatile so there is no guarantee 
previous stores will be visible before a read.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


Review Request 36242: Patch for KAFKA-2311

2015-07-06 Thread Tim Brooks

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36242/
---

Review request for kafka.


Bugs: KAFKA-2311
https://issues.apache.org/jira/browse/KAFKA-2311


Repository: kafka


Description
---

Make closed flag atomic on consumer


Diffs
-

  clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java 
1f0e51557c4569f0980b72652846b250d00e05d6 

Diff: https://reviews.apache.org/r/36242/diff/


Testing
---


Thanks,

Tim Brooks



[jira] [Updated] (KAFKA-2311) Consumer's ensureNotClosed method not thread safe

2015-07-06 Thread Tim Brooks (JIRA)

 [ 
https://issues.apache.org/jira/browse/KAFKA-2311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tim Brooks updated KAFKA-2311:
--
Attachment: KAFKA-2311.patch

> Consumer's ensureNotClosed method not thread safe
> -
>
> Key: KAFKA-2311
> URL: https://issues.apache.org/jira/browse/KAFKA-2311
> Project: Kafka
>  Issue Type: Bug
>  Components: clients
>Reporter: Tim Brooks
> Attachments: KAFKA-2311.patch
>
>
> When a call is to the consumer is made, the first check is to see that the 
> consumer is not closed. This variable is not volatile so there is no 
> guarantee previous stores will be visible before a read.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (KAFKA-2311) Consumer's ensureNotClosed method not thread safe

2015-07-06 Thread Tim Brooks (JIRA)

[ 
https://issues.apache.org/jira/browse/KAFKA-2311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14616155#comment-14616155
 ] 

Tim Brooks commented on KAFKA-2311:
---

Created reviewboard https://reviews.apache.org/r/36242/diff/
 against branch origin/trunk

> Consumer's ensureNotClosed method not thread safe
> -
>
> Key: KAFKA-2311
> URL: https://issues.apache.org/jira/browse/KAFKA-2311
> Project: Kafka
>  Issue Type: Bug
>  Components: clients
>Reporter: Tim Brooks
> Attachments: KAFKA-2311.patch
>
>
> When a call is to the consumer is made, the first check is to see that the 
> consumer is not closed. This variable is not volatile so there is no 
> guarantee previous stores will be visible before a read.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (KAFKA-2311) Consumer's ensureNotClosed method not thread safe

2015-07-06 Thread Tim Brooks (JIRA)

 [ 
https://issues.apache.org/jira/browse/KAFKA-2311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tim Brooks updated KAFKA-2311:
--
Assignee: Tim Brooks
  Status: Patch Available  (was: Open)

> Consumer's ensureNotClosed method not thread safe
> -
>
> Key: KAFKA-2311
> URL: https://issues.apache.org/jira/browse/KAFKA-2311
> Project: Kafka
>  Issue Type: Bug
>  Components: clients
>Reporter: Tim Brooks
>Assignee: Tim Brooks
> Attachments: KAFKA-2311.patch
>
>
> When a call is to the consumer is made, the first check is to see that the 
> consumer is not closed. This variable is not volatile so there is no 
> guarantee previous stores will be visible before a read.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (KAFKA-2129) Consumer could make multiple concurrent metadata requests

2015-07-06 Thread Guozhang Wang (JIRA)

 [ 
https://issues.apache.org/jira/browse/KAFKA-2129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang updated KAFKA-2129:
-
Resolution: Fixed
Status: Resolved  (was: Patch Available)

> Consumer could make multiple concurrent metadata requests
> -
>
> Key: KAFKA-2129
> URL: https://issues.apache.org/jira/browse/KAFKA-2129
> Project: Kafka
>  Issue Type: Bug
>  Components: clients
>Reporter: Tim Brooks
>Assignee: Tim Brooks
> Attachments: KAFKA-2129.patch
>
>
> The NetworkClient's metadataFetchInProgress is neither volatile nor atomic. 
> This protects against multiple metadata requests being made and is read on 
> poll() on the NetworkClient. It is written to when a request is initiated.
> This is fine for the producer. Which seems to have one thread writing. The 
> KafkaConsumer's poll()  method is synchronized, so there will not be more 
> than one writer entering from there. However, the NetworkClient's poll() 
> method is also accessed on the Consumer's partitionsFor() method. Which could 
> be access by a separate thread.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (KAFKA-2102) Remove unnecessary synchronization when managing metadata

2015-07-06 Thread Tim Brooks (JIRA)

[ 
https://issues.apache.org/jira/browse/KAFKA-2102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14616158#comment-14616158
 ] 

Tim Brooks commented on KAFKA-2102:
---

I can probably take another look at this later this week and see if it makes a 
difference in the cases you raised.

> Remove unnecessary synchronization when managing metadata
> -
>
> Key: KAFKA-2102
> URL: https://issues.apache.org/jira/browse/KAFKA-2102
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Tim Brooks
>Assignee: Tim Brooks
> Attachments: KAFKA-2102.patch, KAFKA-2102_2015-04-08_00:20:33.patch, 
> KAFKA-2102_2015-04-15_19:55:45.patch, KAFKA-2102_2015-04-26_17:25:21.patch, 
> eight-threads-patch.txt, eight-threads-trunk.txt, five-threads-patch.txt, 
> five-threads-trunk.txt
>
>
> Usage of the org.apache.kafka.clients.Metadata class is synchronized. It 
> seems like the current functionality could be maintained without 
> synchronizing the whole class.
> I have been working on improving this by moving to finer grained locks and 
> using atomic operations. My initial benchmarking of the producer is that this 
> will improve latency (using HDRHistogram) on submitting messages.
> I have produced an initial patch. I do not necessarily believe this is 
> complete. And I want to definitely produce some more benchmarks. However, I 
> wanted to get early feedback because this change could be deceptively tricky.
> I am interested in knowing if this is:
> 1. Something that is of interest to the maintainers/community.
> 2. Along the right track
> 3. If there are any gotchas that make my current approach naive.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


Re: [ANNOUNCE] New Committer

2015-07-06 Thread Jay Kreps
Congrats, welcome to the team!

-Jay

On Mon, Jul 6, 2015 at 6:08 PM, Joe Stein  wrote:

> I am pleased to announce that the Apache Kafka PMC has voted to invite Gwen
> Shapira as a committer and Gwen has accepted.
>
> Please join me on welcoming and congratulating Gwen.
>
> Thanks for the contribution both in the project (code, email, etc, etc,
> etc) and in throughout the community too(other projects, conferences, etc,
> etc, etc). I look forward to your continued contributions and much more to
> come!
>
> ~ Joe Stein
> - - - - - - - - - - - - - - - - - - -
>  [image: Logo-Black.jpg]
>   http://www.elodina.net
> http://www.stealth.ly
> - - - - - - - - - - - - - - - - - - -
>


Re: [ANNOUNCE] New Committer

2015-07-06 Thread Jun Rao
Gwen,

Congratulations!

Jun

On Mon, Jul 6, 2015 at 6:08 PM, Joe Stein  wrote:

> I am pleased to announce that the Apache Kafka PMC has voted to invite Gwen
> Shapira as a committer and Gwen has accepted.
>
> Please join me on welcoming and congratulating Gwen.
>
> Thanks for the contribution both in the project (code, email, etc, etc,
> etc) and in throughout the community too(other projects, conferences, etc,
> etc, etc). I look forward to your continued contributions and much more to
> come!
>
> ~ Joe Stein
> - - - - - - - - - - - - - - - - - - -
>  [image: Logo-Black.jpg]
>   http://www.elodina.net
> http://www.stealth.ly
> - - - - - - - - - - - - - - - - - - -
>


[jira] [Created] (KAFKA-2312) Use AtomicLong opposed to AtomicReference to store currentThread in consumer

2015-07-06 Thread Tim Brooks (JIRA)
Tim Brooks created KAFKA-2312:
-

 Summary: Use AtomicLong opposed to AtomicReference to store 
currentThread in consumer
 Key: KAFKA-2312
 URL: https://issues.apache.org/jira/browse/KAFKA-2312
 Project: Kafka
  Issue Type: Improvement
  Components: clients
Reporter: Tim Brooks
Priority: Minor


When a thread id is returned by Thread.currentThread().getId() it is a 
primitive. Storing it in an AtomicReference requires boxing and additional 
indirection.

An AtomicLong seems more natural to store a long. 

The current implementation relies on knowing that null means no owner. Since 
thread ids are always positive (specified in javadoc), it is possible to create 
a constant NO_CURRENT_THREAD for -1. Which allows the usage of an AtomicLong 
and makes the functionality explicit.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (KAFKA-2312) Use AtomicLong opposed to AtomicReference to store currentThread in consumer

2015-07-06 Thread Tim Brooks (JIRA)

 [ 
https://issues.apache.org/jira/browse/KAFKA-2312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tim Brooks updated KAFKA-2312:
--
Attachment: KAFKA-2312.patch

> Use AtomicLong opposed to AtomicReference to store currentThread in consumer
> 
>
> Key: KAFKA-2312
> URL: https://issues.apache.org/jira/browse/KAFKA-2312
> Project: Kafka
>  Issue Type: Improvement
>  Components: clients
>Reporter: Tim Brooks
>Priority: Minor
> Attachments: KAFKA-2312.patch
>
>
> When a thread id is returned by Thread.currentThread().getId() it is a 
> primitive. Storing it in an AtomicReference requires boxing and additional 
> indirection.
> An AtomicLong seems more natural to store a long. 
> The current implementation relies on knowing that null means no owner. Since 
> thread ids are always positive (specified in javadoc), it is possible to 
> create a constant NO_CURRENT_THREAD for -1. Which allows the usage of an 
> AtomicLong and makes the functionality explicit.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (KAFKA-2312) Use AtomicLong opposed to AtomicReference to store currentThread in consumer

2015-07-06 Thread Tim Brooks (JIRA)

 [ 
https://issues.apache.org/jira/browse/KAFKA-2312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tim Brooks updated KAFKA-2312:
--
Assignee: Tim Brooks
  Status: Patch Available  (was: Open)

> Use AtomicLong opposed to AtomicReference to store currentThread in consumer
> 
>
> Key: KAFKA-2312
> URL: https://issues.apache.org/jira/browse/KAFKA-2312
> Project: Kafka
>  Issue Type: Improvement
>  Components: clients
>Reporter: Tim Brooks
>Assignee: Tim Brooks
>Priority: Minor
> Attachments: KAFKA-2312.patch
>
>
> When a thread id is returned by Thread.currentThread().getId() it is a 
> primitive. Storing it in an AtomicReference requires boxing and additional 
> indirection.
> An AtomicLong seems more natural to store a long. 
> The current implementation relies on knowing that null means no owner. Since 
> thread ids are always positive (specified in javadoc), it is possible to 
> create a constant NO_CURRENT_THREAD for -1. Which allows the usage of an 
> AtomicLong and makes the functionality explicit.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


Review Request 36244: Patch for KAFKA-2312

2015-07-06 Thread Tim Brooks

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36244/
---

Review request for kafka.


Bugs: KAFKA-2312
https://issues.apache.org/jira/browse/KAFKA-2312


Repository: kafka


Description
---

Use an atomic long for the 'light lock' opposed to an atomic reference.


Diffs
-

  clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java 
1f0e51557c4569f0980b72652846b250d00e05d6 

Diff: https://reviews.apache.org/r/36244/diff/


Testing
---


Thanks,

Tim Brooks



[jira] [Commented] (KAFKA-2312) Use AtomicLong opposed to AtomicReference to store currentThread in consumer

2015-07-06 Thread Tim Brooks (JIRA)

[ 
https://issues.apache.org/jira/browse/KAFKA-2312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14616179#comment-14616179
 ] 

Tim Brooks commented on KAFKA-2312:
---

Created reviewboard https://reviews.apache.org/r/36244/diff/
 against branch origin/trunk

> Use AtomicLong opposed to AtomicReference to store currentThread in consumer
> 
>
> Key: KAFKA-2312
> URL: https://issues.apache.org/jira/browse/KAFKA-2312
> Project: Kafka
>  Issue Type: Improvement
>  Components: clients
>Reporter: Tim Brooks
>Priority: Minor
> Attachments: KAFKA-2312.patch
>
>
> When a thread id is returned by Thread.currentThread().getId() it is a 
> primitive. Storing it in an AtomicReference requires boxing and additional 
> indirection.
> An AtomicLong seems more natural to store a long. 
> The current implementation relies on knowing that null means no owner. Since 
> thread ids are always positive (specified in javadoc), it is possible to 
> create a constant NO_CURRENT_THREAD for -1. Which allows the usage of an 
> AtomicLong and makes the functionality explicit.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


Re: Review Request 35820: Patch for KAFKA-1367

2015-07-06 Thread Ashish Singh

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/35820/
---

(Updated July 7, 2015, 5:04 a.m.)


Review request for kafka.


Bugs: KAFKA-1367
https://issues.apache.org/jira/browse/KAFKA-1367


Repository: kafka


Description
---

KAFKA-1367: Broker topic metadata not kept in sync with ZooKeeper


Diffs (updated)
-

  core/src/main/scala/kafka/common/TopicAndPartition.scala 
df3db912f5daef6a25b4b2dd2220d2cc3795bce6 
  core/src/main/scala/kafka/controller/KafkaController.scala 
36350579b16027359d237b64699003358704ac6f 
  core/src/main/scala/kafka/utils/ReplicationUtils.scala 
60687332b4c9bee4d4c0851314cfb4b02d5d3489 
  core/src/main/scala/kafka/utils/ZkUtils.scala 
78475e3d5ec477cef00caeaa34ff2d196466be96 
  core/src/test/scala/unit/kafka/integration/TopicMetadataTest.scala 
995b05901491bb0dbf0df210d44bd1d7f66fdc82 
  core/src/test/scala/unit/kafka/utils/ReplicationUtilsTest.scala 
c96c0ffd958d63c09880d436b2e5ae96f51ead36 

Diff: https://reviews.apache.org/r/35820/diff/


Testing
---

Tested on a test cluster with 3 Kafka brokers


Thanks,

Ashish Singh



Re: Review Request 35820: Patch for KAFKA-1367

2015-07-06 Thread Ashish Singh


> On July 7, 2015, 12:36 a.m., Jun Rao wrote:
> > core/src/test/scala/unit/kafka/integration/TopicMetadataTest.scala, line 50
> > 
> >
> > This seems to be specific to testIsrAfterBrokerShutDownAndJoinsBack. 
> > Could we move it there?

KAFKA-972 also uses this. I think its best to have it initialized just once in 
setup. Let me know if you think otherwise.


> On July 7, 2015, 12:36 a.m., Jun Rao wrote:
> > core/src/test/scala/unit/kafka/integration/TopicMetadataTest.scala, line 143
> > 
> >
> > Can this be private?

It should be. Thanks for pointing out.


- Ashish


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/35820/#review90605
---


On July 7, 2015, 5:04 a.m., Ashish Singh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/35820/
> ---
> 
> (Updated July 7, 2015, 5:04 a.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1367
> https://issues.apache.org/jira/browse/KAFKA-1367
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> KAFKA-1367: Broker topic metadata not kept in sync with ZooKeeper
> 
> 
> Diffs
> -
> 
>   core/src/main/scala/kafka/common/TopicAndPartition.scala 
> df3db912f5daef6a25b4b2dd2220d2cc3795bce6 
>   core/src/main/scala/kafka/controller/KafkaController.scala 
> 36350579b16027359d237b64699003358704ac6f 
>   core/src/main/scala/kafka/utils/ReplicationUtils.scala 
> 60687332b4c9bee4d4c0851314cfb4b02d5d3489 
>   core/src/main/scala/kafka/utils/ZkUtils.scala 
> 78475e3d5ec477cef00caeaa34ff2d196466be96 
>   core/src/test/scala/unit/kafka/integration/TopicMetadataTest.scala 
> 995b05901491bb0dbf0df210d44bd1d7f66fdc82 
>   core/src/test/scala/unit/kafka/utils/ReplicationUtilsTest.scala 
> c96c0ffd958d63c09880d436b2e5ae96f51ead36 
> 
> Diff: https://reviews.apache.org/r/35820/diff/
> 
> 
> Testing
> ---
> 
> Tested on a test cluster with 3 Kafka brokers
> 
> 
> Thanks,
> 
> Ashish Singh
> 
>



[jira] [Commented] (KAFKA-1367) Broker topic metadata not kept in sync with ZooKeeper

2015-07-06 Thread Ashish K Singh (JIRA)

[ 
https://issues.apache.org/jira/browse/KAFKA-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14616184#comment-14616184
 ] 

Ashish K Singh commented on KAFKA-1367:
---

Updated reviewboard https://reviews.apache.org/r/35820/
 against branch trunk

> Broker topic metadata not kept in sync with ZooKeeper
> -
>
> Key: KAFKA-1367
> URL: https://issues.apache.org/jira/browse/KAFKA-1367
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.0, 0.8.1
>Reporter: Ryan Berdeen
>Assignee: Ashish K Singh
>  Labels: newbie++
> Fix For: 0.8.3
>
> Attachments: KAFKA-1367.patch, KAFKA-1367.txt, 
> KAFKA-1367_2015-07-01_17:23:14.patch, KAFKA-1367_2015-07-06_22:04:06.patch
>
>
> When a broker is restarted, the topic metadata responses from the brokers 
> will be incorrect (different from ZooKeeper) until a preferred replica leader 
> election.
> In the metadata, it looks like leaders are correctly removed from the ISR 
> when a broker disappears, but followers are not. Then, when a broker 
> reappears, the ISR is never updated.
> I used a variation of the Vagrant setup created by Joe Stein to reproduce 
> this with latest from the 0.8.1 branch: 
> https://github.com/also/kafka/commit/dba36a503a5e22ea039df0f9852560b4fb1e067c



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


Re: Review Request 35820: Patch for KAFKA-1367

2015-07-06 Thread Ashish Singh

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/35820/#review90630
---



core/src/main/scala/kafka/controller/KafkaController.scala (line 42)


Moved to method that uses it.


- Ashish Singh


On July 7, 2015, 5:04 a.m., Ashish Singh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/35820/
> ---
> 
> (Updated July 7, 2015, 5:04 a.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1367
> https://issues.apache.org/jira/browse/KAFKA-1367
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> KAFKA-1367: Broker topic metadata not kept in sync with ZooKeeper
> 
> 
> Diffs
> -
> 
>   core/src/main/scala/kafka/common/TopicAndPartition.scala 
> df3db912f5daef6a25b4b2dd2220d2cc3795bce6 
>   core/src/main/scala/kafka/controller/KafkaController.scala 
> 36350579b16027359d237b64699003358704ac6f 
>   core/src/main/scala/kafka/utils/ReplicationUtils.scala 
> 60687332b4c9bee4d4c0851314cfb4b02d5d3489 
>   core/src/main/scala/kafka/utils/ZkUtils.scala 
> 78475e3d5ec477cef00caeaa34ff2d196466be96 
>   core/src/test/scala/unit/kafka/integration/TopicMetadataTest.scala 
> 995b05901491bb0dbf0df210d44bd1d7f66fdc82 
>   core/src/test/scala/unit/kafka/utils/ReplicationUtilsTest.scala 
> c96c0ffd958d63c09880d436b2e5ae96f51ead36 
> 
> Diff: https://reviews.apache.org/r/35820/diff/
> 
> 
> Testing
> ---
> 
> Tested on a test cluster with 3 Kafka brokers
> 
> 
> Thanks,
> 
> Ashish Singh
> 
>



[jira] [Updated] (KAFKA-1367) Broker topic metadata not kept in sync with ZooKeeper

2015-07-06 Thread Ashish K Singh (JIRA)

 [ 
https://issues.apache.org/jira/browse/KAFKA-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashish K Singh updated KAFKA-1367:
--
Status: Patch Available  (was: In Progress)

> Broker topic metadata not kept in sync with ZooKeeper
> -
>
> Key: KAFKA-1367
> URL: https://issues.apache.org/jira/browse/KAFKA-1367
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.0, 0.8.1
>Reporter: Ryan Berdeen
>Assignee: Ashish K Singh
>  Labels: newbie++
> Fix For: 0.8.3
>
> Attachments: KAFKA-1367.patch, KAFKA-1367.txt, 
> KAFKA-1367_2015-07-01_17:23:14.patch, KAFKA-1367_2015-07-06_22:04:06.patch
>
>
> When a broker is restarted, the topic metadata responses from the brokers 
> will be incorrect (different from ZooKeeper) until a preferred replica leader 
> election.
> In the metadata, it looks like leaders are correctly removed from the ISR 
> when a broker disappears, but followers are not. Then, when a broker 
> reappears, the ISR is never updated.
> I used a variation of the Vagrant setup created by Joe Stein to reproduce 
> this with latest from the 0.8.1 branch: 
> https://github.com/also/kafka/commit/dba36a503a5e22ea039df0f9852560b4fb1e067c



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (KAFKA-1367) Broker topic metadata not kept in sync with ZooKeeper

2015-07-06 Thread Ashish K Singh (JIRA)

 [ 
https://issues.apache.org/jira/browse/KAFKA-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashish K Singh updated KAFKA-1367:
--
Attachment: KAFKA-1367_2015-07-06_22:04:06.patch

> Broker topic metadata not kept in sync with ZooKeeper
> -
>
> Key: KAFKA-1367
> URL: https://issues.apache.org/jira/browse/KAFKA-1367
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.0, 0.8.1
>Reporter: Ryan Berdeen
>Assignee: Ashish K Singh
>  Labels: newbie++
> Fix For: 0.8.3
>
> Attachments: KAFKA-1367.patch, KAFKA-1367.txt, 
> KAFKA-1367_2015-07-01_17:23:14.patch, KAFKA-1367_2015-07-06_22:04:06.patch
>
>
> When a broker is restarted, the topic metadata responses from the brokers 
> will be incorrect (different from ZooKeeper) until a preferred replica leader 
> election.
> In the metadata, it looks like leaders are correctly removed from the ISR 
> when a broker disappears, but followers are not. Then, when a broker 
> reappears, the ISR is never updated.
> I used a variation of the Vagrant setup created by Joe Stein to reproduce 
> this with latest from the 0.8.1 branch: 
> https://github.com/also/kafka/commit/dba36a503a5e22ea039df0f9852560b4fb1e067c



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


Re: [jira] [Commented] (KAFKA-2132) Move Log4J appender to a separate module

2015-07-06 Thread Jun Rao
Stevo,

I don't see duplicated slf4j-log4j12 jars under core/build/dependant-libs
after a clean build in trunk. If this is still an issue, could you file a
jira and describe how to reproduce this?

Thanks,

Jun

On Fri, Jun 26, 2015 at 2:19 PM, Stevo Slavić  wrote:

> Are changes for KAFKA-2132 ticket supposed also to fix bug that core
> dependent libraries (core/build/dependant-libs) for all different supported
> Scala version, contain two versions of slf4j-log4j12
> (slf4j-log4j12-1.6.1.jar leaking from zookeeper 3.4.6 dependency, and test
> scoped slf4j-log4j12-1.7.6.jar dependency - latter is explicitly added for
> some reason in copyDependantLibs task, but it does not override
> slf4j-log4j12 leak from zookeeper) ?
>
> It's the source of annoying:
>
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
>
> [jar:file:/Users/foo/kafka/core/build/dependant-libs-2.10.5/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
>
> [jar:file:/Users/foo/kafka/core/build/dependant-libs-2.10.5/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
>
> whenever a script like bin/kafka-topics.sh is run.
>
> Or should separate ticket be filed for this issue?
>
> Kind regard,
> Stevo Slavic.
>
> On Wed, Jun 24, 2015 at 7:26 PM, Ashish K Singh (JIRA) 
> wrote:
>
> >
> > [
> >
> https://issues.apache.org/jira/browse/KAFKA-2132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14599778#comment-14599778
> > ]
> >
> > Ashish K Singh commented on KAFKA-2132:
> > ---
> >
> > Updated reviewboard https://reviews.apache.org/r/33614/
> >  against branch trunk
> >
> > > Move Log4J appender to a separate module
> > > 
> > >
> > > Key: KAFKA-2132
> > > URL: https://issues.apache.org/jira/browse/KAFKA-2132
> > > Project: Kafka
> > >  Issue Type: Improvement
> > >Reporter: Gwen Shapira
> > >Assignee: Ashish K Singh
> > > Attachments: KAFKA-2132.patch,
> > KAFKA-2132_2015-04-27_19:59:46.patch,
> KAFKA-2132_2015-04-30_12:22:02.patch,
> > KAFKA-2132_2015-04-30_15:53:17.patch,
> KAFKA-2132_2015-06-13_21:18:59.patch,
> > KAFKA-2132_2015-06-24_10:19:56.patch,
> KAFKA-2132_2015-06-24_10:25:43.patch
> > >
> > >
> > > Log4j appender is just a producer.
> > > Since we have a new producer in the clients module, no need to keep
> > Log4J appender in "core" and force people to package all of Kafka with
> > their apps.
> > > Lets move the Log4jAppender to clients module.
> >
> >
> >
> > --
> > This message was sent by Atlassian JIRA
> > (v6.3.4#6332)
> >
>


Re: Review Request 36242: Patch for KAFKA-2311

2015-07-06 Thread Ewen Cheslack-Postava

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36242/#review90644
---



clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java 
(line 987)


This check could even be removed -- if acquire() succeeded, then this would 
only return if there was incorrect multi-threaded access.

Actually, now that I think about it more, it seems like maybe we changed 
the semanitcs of close() a bit. Clearly it was previously intended to be safe 
to call multiple times, but acquire() will throw if it's already been closed. 
KafkaProducer doesn't have the same type of check, although I'm not certain 
what happens if you invoke close twice there (and it also has a different 
threading model).

I think it might be better to not allow double closing this, so I think I'd 
just remove the check and leave the possibility that acquire() causes close() 
to throw an exception.



clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java 
(line 1345)


Only related to this patch because I'm looking at ensureNotClosed -- it's 
only used in one place and is a trivial check. We could just inline it.


- Ewen Cheslack-Postava


On July 7, 2015, 4:29 a.m., Tim Brooks wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/36242/
> ---
> 
> (Updated July 7, 2015, 4:29 a.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-2311
> https://issues.apache.org/jira/browse/KAFKA-2311
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> Make closed flag atomic on consumer
> 
> 
> Diffs
> -
> 
>   clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java 
> 1f0e51557c4569f0980b72652846b250d00e05d6 
> 
> Diff: https://reviews.apache.org/r/36242/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Tim Brooks
> 
>



Re: [ANNOUNCE] New Committer

2015-07-06 Thread gharatmayuresh15
Congratulations! Gwen

Mayuresh

Sent from my iPhone

> On Jul 6, 2015, at 9:46 PM, Jun Rao  wrote:
> 
> Gwen,
> 
> Congratulations!
> 
> Jun
> 
>> On Mon, Jul 6, 2015 at 6:08 PM, Joe Stein  wrote:
>> 
>> I am pleased to announce that the Apache Kafka PMC has voted to invite Gwen
>> Shapira as a committer and Gwen has accepted.
>> 
>> Please join me on welcoming and congratulating Gwen.
>> 
>> Thanks for the contribution both in the project (code, email, etc, etc,
>> etc) and in throughout the community too(other projects, conferences, etc,
>> etc, etc). I look forward to your continued contributions and much more to
>> come!
>> 
>> ~ Joe Stein
>> - - - - - - - - - - - - - - - - - - -
>> [image: Logo-Black.jpg]
>>  http://www.elodina.net
>>http://www.stealth.ly
>> - - - - - - - - - - - - - - - - - - -
>> 


Re: Review Request 36244: Patch for KAFKA-2312

2015-07-06 Thread Ewen Cheslack-Postava

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36244/#review90651
---

Ship it!


Performance-wise, it's a micro-optimization that probably doesn't have any real 
impact in practical code, but may be a bit more intuitive.

- Ewen Cheslack-Postava


On July 7, 2015, 5 a.m., Tim Brooks wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/36244/
> ---
> 
> (Updated July 7, 2015, 5 a.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-2312
> https://issues.apache.org/jira/browse/KAFKA-2312
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> Use an atomic long for the 'light lock' opposed to an atomic reference.
> 
> 
> Diffs
> -
> 
>   clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java 
> 1f0e51557c4569f0980b72652846b250d00e05d6 
> 
> Diff: https://reviews.apache.org/r/36244/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Tim Brooks
> 
>



[jira] [Commented] (KAFKA-972) MetadataRequest returns stale list of brokers

2015-07-06 Thread Ashish K Singh (JIRA)

[ 
https://issues.apache.org/jira/browse/KAFKA-972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14616238#comment-14616238
 ] 

Ashish K Singh commented on KAFKA-972:
--

Updated reviewboard https://reviews.apache.org/r/36030/
 against branch trunk

> MetadataRequest returns stale list of brokers
> -
>
> Key: KAFKA-972
> URL: https://issues.apache.org/jira/browse/KAFKA-972
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.8.0
>Reporter: Vinicius Carvalho
>Assignee: Ashish K Singh
> Attachments: BrokerMetadataTest.scala, KAFKA-972.patch, 
> KAFKA-972_2015-06-30_18:42:13.patch, KAFKA-972_2015-07-01_01:36:56.patch, 
> KAFKA-972_2015-07-01_01:42:42.patch, KAFKA-972_2015-07-01_08:06:03.patch, 
> KAFKA-972_2015-07-06_23:07:34.patch
>
>
> When we issue an metadatarequest towards the cluster, the list of brokers is 
> stale. I mean, even when a broker is down, it's returned back to the client. 
> The following are examples of two invocations one with both brokers online 
> and the second with a broker down:
> {
> "brokers": [
> {
> "nodeId": 0,
> "host": "10.139.245.106",
> "port": 9092,
> "byteLength": 24
> },
> {
> "nodeId": 1,
> "host": "localhost",
> "port": 9093,
> "byteLength": 19
> }
> ],
> "topicMetadata": [
> {
> "topicErrorCode": 0,
> "topicName": "foozbar",
> "partitions": [
> {
> "replicas": [
> 0
> ],
> "isr": [
> 0
> ],
> "partitionErrorCode": 0,
> "partitionId": 0,
> "leader": 0,
> "byteLength": 26
> },
> {
> "replicas": [
> 1
> ],
> "isr": [
> 1
> ],
> "partitionErrorCode": 0,
> "partitionId": 1,
> "leader": 1,
> "byteLength": 26
> },
> {
> "replicas": [
> 0
> ],
> "isr": [
> 0
> ],
> "partitionErrorCode": 0,
> "partitionId": 2,
> "leader": 0,
> "byteLength": 26
> },
> {
> "replicas": [
> 1
> ],
> "isr": [
> 1
> ],
> "partitionErrorCode": 0,
> "partitionId": 3,
> "leader": 1,
> "byteLength": 26
> },
> {
> "replicas": [
> 0
> ],
> "isr": [
> 0
> ],
> "partitionErrorCode": 0,
> "partitionId": 4,
> "leader": 0,
> "byteLength": 26
> }
> ],
> "byteLength": 145
> }
> ],
> "responseSize": 200,
> "correlationId": -1000
> }
> {
> "brokers": [
> {
> "nodeId": 0,
> "host": "10.139.245.106",
> "port": 9092,
> "byteLength": 24
> },
> {
> "nodeId": 1,
> "host": "localhost",
> "port": 9093,
> "byteLength": 19
> }
> ],
> "topicMetadata": [
> {
> "topicErrorCode": 0,
> "topicName": "foozbar",
> "partitions": [
> {
> "replicas": [
> 0
> ],
> "isr": [],
> "partitionErrorCode": 5,
> "partitionId": 0,
> "leader": -1,
> "byteLength": 22
> },
> {
> "replicas": [
> 1
> ],
> "isr": [
> 1
> ],
> "partitionErrorCode": 0,
> "partitionId": 1,
> "leader": 1,
> "byteLength": 26
> },
> {
> "replicas": [
> 0
>  

Re: Review Request 36030: Patch for KAFKA-972

2015-07-06 Thread Ashish Singh


> On July 6, 2015, 9:47 p.m., Jun Rao wrote:
> > Thanks for the patch. A few more minor comments blow.

Thanks for the review again Jun!


> On July 6, 2015, 9:47 p.m., Jun Rao wrote:
> > core/src/test/scala/unit/kafka/integration/TopicMetadataTest.scala, line 147
> > 
> >
> > This can be private, right?

It should be. Thanks for pointing out.


> On July 6, 2015, 9:47 p.m., Jun Rao wrote:
> > core/src/test/scala/unit/kafka/integration/TopicMetadataTest.scala, lines 
> > 148-156
> > 
> >
> > This seems redundant given the code in 155 to 163. We can probaby just 
> > assert the broker size on topicMetadata after line 152.

True


- Ashish


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36030/#review90579
---


On July 7, 2015, 6:07 a.m., Ashish Singh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/36030/
> ---
> 
> (Updated July 7, 2015, 6:07 a.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-972
> https://issues.apache.org/jira/browse/KAFKA-972
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> KAFKA-972: MetadataRequest returns stale list of brokers
> 
> 
> Diffs
> -
> 
>   core/src/main/scala/kafka/controller/KafkaController.scala 
> 36350579b16027359d237b64699003358704ac6f 
>   core/src/test/scala/unit/kafka/integration/TopicMetadataTest.scala 
> 995b05901491bb0dbf0df210d44bd1d7f66fdc82 
> 
> Diff: https://reviews.apache.org/r/36030/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Ashish Singh
> 
>



[jira] [Updated] (KAFKA-972) MetadataRequest returns stale list of brokers

2015-07-06 Thread Ashish K Singh (JIRA)

 [ 
https://issues.apache.org/jira/browse/KAFKA-972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashish K Singh updated KAFKA-972:
-
Attachment: KAFKA-972_2015-07-06_23:07:34.patch

> MetadataRequest returns stale list of brokers
> -
>
> Key: KAFKA-972
> URL: https://issues.apache.org/jira/browse/KAFKA-972
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.8.0
>Reporter: Vinicius Carvalho
>Assignee: Ashish K Singh
> Attachments: BrokerMetadataTest.scala, KAFKA-972.patch, 
> KAFKA-972_2015-06-30_18:42:13.patch, KAFKA-972_2015-07-01_01:36:56.patch, 
> KAFKA-972_2015-07-01_01:42:42.patch, KAFKA-972_2015-07-01_08:06:03.patch, 
> KAFKA-972_2015-07-06_23:07:34.patch
>
>
> When we issue an metadatarequest towards the cluster, the list of brokers is 
> stale. I mean, even when a broker is down, it's returned back to the client. 
> The following are examples of two invocations one with both brokers online 
> and the second with a broker down:
> {
> "brokers": [
> {
> "nodeId": 0,
> "host": "10.139.245.106",
> "port": 9092,
> "byteLength": 24
> },
> {
> "nodeId": 1,
> "host": "localhost",
> "port": 9093,
> "byteLength": 19
> }
> ],
> "topicMetadata": [
> {
> "topicErrorCode": 0,
> "topicName": "foozbar",
> "partitions": [
> {
> "replicas": [
> 0
> ],
> "isr": [
> 0
> ],
> "partitionErrorCode": 0,
> "partitionId": 0,
> "leader": 0,
> "byteLength": 26
> },
> {
> "replicas": [
> 1
> ],
> "isr": [
> 1
> ],
> "partitionErrorCode": 0,
> "partitionId": 1,
> "leader": 1,
> "byteLength": 26
> },
> {
> "replicas": [
> 0
> ],
> "isr": [
> 0
> ],
> "partitionErrorCode": 0,
> "partitionId": 2,
> "leader": 0,
> "byteLength": 26
> },
> {
> "replicas": [
> 1
> ],
> "isr": [
> 1
> ],
> "partitionErrorCode": 0,
> "partitionId": 3,
> "leader": 1,
> "byteLength": 26
> },
> {
> "replicas": [
> 0
> ],
> "isr": [
> 0
> ],
> "partitionErrorCode": 0,
> "partitionId": 4,
> "leader": 0,
> "byteLength": 26
> }
> ],
> "byteLength": 145
> }
> ],
> "responseSize": 200,
> "correlationId": -1000
> }
> {
> "brokers": [
> {
> "nodeId": 0,
> "host": "10.139.245.106",
> "port": 9092,
> "byteLength": 24
> },
> {
> "nodeId": 1,
> "host": "localhost",
> "port": 9093,
> "byteLength": 19
> }
> ],
> "topicMetadata": [
> {
> "topicErrorCode": 0,
> "topicName": "foozbar",
> "partitions": [
> {
> "replicas": [
> 0
> ],
> "isr": [],
> "partitionErrorCode": 5,
> "partitionId": 0,
> "leader": -1,
> "byteLength": 22
> },
> {
> "replicas": [
> 1
> ],
> "isr": [
> 1
> ],
> "partitionErrorCode": 0,
> "partitionId": 1,
> "leader": 1,
> "byteLength": 26
> },
> {
> "replicas": [
> 0
> ],
> "isr": [],
> "partitionError

[jira] [Updated] (KAFKA-972) MetadataRequest returns stale list of brokers

2015-07-06 Thread Ashish K Singh (JIRA)

 [ 
https://issues.apache.org/jira/browse/KAFKA-972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashish K Singh updated KAFKA-972:
-
Status: Patch Available  (was: In Progress)

> MetadataRequest returns stale list of brokers
> -
>
> Key: KAFKA-972
> URL: https://issues.apache.org/jira/browse/KAFKA-972
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.8.0
>Reporter: Vinicius Carvalho
>Assignee: Ashish K Singh
> Attachments: BrokerMetadataTest.scala, KAFKA-972.patch, 
> KAFKA-972_2015-06-30_18:42:13.patch, KAFKA-972_2015-07-01_01:36:56.patch, 
> KAFKA-972_2015-07-01_01:42:42.patch, KAFKA-972_2015-07-01_08:06:03.patch, 
> KAFKA-972_2015-07-06_23:07:34.patch
>
>
> When we issue an metadatarequest towards the cluster, the list of brokers is 
> stale. I mean, even when a broker is down, it's returned back to the client. 
> The following are examples of two invocations one with both brokers online 
> and the second with a broker down:
> {
> "brokers": [
> {
> "nodeId": 0,
> "host": "10.139.245.106",
> "port": 9092,
> "byteLength": 24
> },
> {
> "nodeId": 1,
> "host": "localhost",
> "port": 9093,
> "byteLength": 19
> }
> ],
> "topicMetadata": [
> {
> "topicErrorCode": 0,
> "topicName": "foozbar",
> "partitions": [
> {
> "replicas": [
> 0
> ],
> "isr": [
> 0
> ],
> "partitionErrorCode": 0,
> "partitionId": 0,
> "leader": 0,
> "byteLength": 26
> },
> {
> "replicas": [
> 1
> ],
> "isr": [
> 1
> ],
> "partitionErrorCode": 0,
> "partitionId": 1,
> "leader": 1,
> "byteLength": 26
> },
> {
> "replicas": [
> 0
> ],
> "isr": [
> 0
> ],
> "partitionErrorCode": 0,
> "partitionId": 2,
> "leader": 0,
> "byteLength": 26
> },
> {
> "replicas": [
> 1
> ],
> "isr": [
> 1
> ],
> "partitionErrorCode": 0,
> "partitionId": 3,
> "leader": 1,
> "byteLength": 26
> },
> {
> "replicas": [
> 0
> ],
> "isr": [
> 0
> ],
> "partitionErrorCode": 0,
> "partitionId": 4,
> "leader": 0,
> "byteLength": 26
> }
> ],
> "byteLength": 145
> }
> ],
> "responseSize": 200,
> "correlationId": -1000
> }
> {
> "brokers": [
> {
> "nodeId": 0,
> "host": "10.139.245.106",
> "port": 9092,
> "byteLength": 24
> },
> {
> "nodeId": 1,
> "host": "localhost",
> "port": 9093,
> "byteLength": 19
> }
> ],
> "topicMetadata": [
> {
> "topicErrorCode": 0,
> "topicName": "foozbar",
> "partitions": [
> {
> "replicas": [
> 0
> ],
> "isr": [],
> "partitionErrorCode": 5,
> "partitionId": 0,
> "leader": -1,
> "byteLength": 22
> },
> {
> "replicas": [
> 1
> ],
> "isr": [
> 1
> ],
> "partitionErrorCode": 0,
> "partitionId": 1,
> "leader": 1,
> "byteLength": 26
> },
> {
> "replicas": [
> 0
> ],
> "isr": [],
> "partitionErrorCode

Re: Review Request 36030: Patch for KAFKA-972

2015-07-06 Thread Ashish Singh

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36030/
---

(Updated July 7, 2015, 6:07 a.m.)


Review request for kafka.


Bugs: KAFKA-972
https://issues.apache.org/jira/browse/KAFKA-972


Repository: kafka


Description
---

KAFKA-972: MetadataRequest returns stale list of brokers


Diffs (updated)
-

  core/src/main/scala/kafka/controller/KafkaController.scala 
36350579b16027359d237b64699003358704ac6f 
  core/src/test/scala/unit/kafka/integration/TopicMetadataTest.scala 
995b05901491bb0dbf0df210d44bd1d7f66fdc82 

Diff: https://reviews.apache.org/r/36030/diff/


Testing
---


Thanks,

Ashish Singh



Re: [jira] [Commented] (KAFKA-2132) Move Log4J appender to a separate module

2015-07-06 Thread Stevo Slavić
Hello Jun,

I can easily reproduce the issue with previous commit (RAT).
While on latest trunk branch:

$ git checkout HEAD^
$ gradle clean
$ gradle copyDependantLibs
$ ls -lart core/build/dependant-libs-2.10.5/

lists slf4j-api-1.7.6.jar and slf4j-api-1.6.1.jar

Not sure exactly which modification in last commit did it, but issue is no
longer there as of last commit.

Thanks!

Kind regards,
Stevo Slavic.

On Tue, Jul 7, 2015 at 7:10 AM, Jun Rao  wrote:

> Stevo,
>
> I don't see duplicated slf4j-log4j12 jars under core/build/dependant-libs
> after a clean build in trunk. If this is still an issue, could you file a
> jira and describe how to reproduce this?
>
> Thanks,
>
> Jun
>
> On Fri, Jun 26, 2015 at 2:19 PM, Stevo Slavić  wrote:
>
> > Are changes for KAFKA-2132 ticket supposed also to fix bug that core
> > dependent libraries (core/build/dependant-libs) for all different
> supported
> > Scala version, contain two versions of slf4j-log4j12
> > (slf4j-log4j12-1.6.1.jar leaking from zookeeper 3.4.6 dependency, and
> test
> > scoped slf4j-log4j12-1.7.6.jar dependency - latter is explicitly added
> for
> > some reason in copyDependantLibs task, but it does not override
> > slf4j-log4j12 leak from zookeeper) ?
> >
> > It's the source of annoying:
> >
> > SLF4J: Class path contains multiple SLF4J bindings.
> > SLF4J: Found binding in
> >
> >
> [jar:file:/Users/foo/kafka/core/build/dependant-libs-2.10.5/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> > SLF4J: Found binding in
> >
> >
> [jar:file:/Users/foo/kafka/core/build/dependant-libs-2.10.5/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> > explanation.
> > SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> >
> > whenever a script like bin/kafka-topics.sh is run.
> >
> > Or should separate ticket be filed for this issue?
> >
> > Kind regard,
> > Stevo Slavic.
> >
> > On Wed, Jun 24, 2015 at 7:26 PM, Ashish K Singh (JIRA) 
> > wrote:
> >
> > >
> > > [
> > >
> >
> https://issues.apache.org/jira/browse/KAFKA-2132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14599778#comment-14599778
> > > ]
> > >
> > > Ashish K Singh commented on KAFKA-2132:
> > > ---
> > >
> > > Updated reviewboard https://reviews.apache.org/r/33614/
> > >  against branch trunk
> > >
> > > > Move Log4J appender to a separate module
> > > > 
> > > >
> > > > Key: KAFKA-2132
> > > > URL:
> https://issues.apache.org/jira/browse/KAFKA-2132
> > > > Project: Kafka
> > > >  Issue Type: Improvement
> > > >Reporter: Gwen Shapira
> > > >Assignee: Ashish K Singh
> > > > Attachments: KAFKA-2132.patch,
> > > KAFKA-2132_2015-04-27_19:59:46.patch,
> > KAFKA-2132_2015-04-30_12:22:02.patch,
> > > KAFKA-2132_2015-04-30_15:53:17.patch,
> > KAFKA-2132_2015-06-13_21:18:59.patch,
> > > KAFKA-2132_2015-06-24_10:19:56.patch,
> > KAFKA-2132_2015-06-24_10:25:43.patch
> > > >
> > > >
> > > > Log4j appender is just a producer.
> > > > Since we have a new producer in the clients module, no need to keep
> > > Log4J appender in "core" and force people to package all of Kafka with
> > > their apps.
> > > > Lets move the Log4jAppender to clients module.
> > >
> > >
> > >
> > > --
> > > This message was sent by Atlassian JIRA
> > > (v6.3.4#6332)
> > >
> >
>