Re: Review Request 34492: Patch for KAFKA-2210

2015-06-11 Thread Dapeng Sun
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/34492/#review87532 --- core/src/main/scala/kafka/server/KafkaConfig.scala

offset storage as kafka with zookeeper 3.4.6

2015-06-11 Thread Kris K
I am trying to migrate the offset storage to kafka (3 brokers of version 0.8.2.1) using the consumer property offsets.storage=kafka. I noticed that a new topic, __consumer_offsets got created. But nothing is being written to this topic, while the consumer offsets continue to reside on zookeeper.

[jira] [Created] (KAFKA-2263) Update "Is it possible to delete a topic" wiki FAQ answer

2015-06-11 Thread Stevo Slavic (JIRA)
Stevo Slavic created KAFKA-2263: --- Summary: Update "Is it possible to delete a topic" wiki FAQ answer Key: KAFKA-2263 URL: https://issues.apache.org/jira/browse/KAFKA-2263 Project: Kafka Issue T

Re: offset storage as kafka with zookeeper 3.4.6

2015-06-11 Thread Joel Koshy
> Is it mandatory to use the zookeeper that comes with kafka for offset > storage to be migrated to kafka? If you want to "move" offsets from zookeeper to Kafka then yes you need to have a phase where all consumers in your group set dual commit to true. If you are starting a fresh consumer group th

[jira] [Created] (KAFKA-2264) SESSION_TIMEOUT_MS_CONFIG in ConsumerConfig should be int

2015-06-11 Thread Jun Rao (JIRA)
Jun Rao created KAFKA-2264: -- Summary: SESSION_TIMEOUT_MS_CONFIG in ConsumerConfig should be int Key: KAFKA-2264 URL: https://issues.apache.org/jira/browse/KAFKA-2264 Project: Kafka Issue Type: Bug

Re: offset storage as kafka with zookeeper 3.4.6

2015-06-11 Thread Kris K
>If you want to "move" offsets from zookeeper to Kafka then yes you >need to have a phase where all consumers in your group set dual commit >to true. If you are starting a fresh consumer group then you can >turn off dual-commit. I followed these steps to move the offsets from zookeeper to kafka: 1.

Re: Review Request 35201: Fix KAFKA-2253

2015-06-11 Thread Jiangjie Qin
> On June 11, 2015, 1:07 a.m., Jun Rao wrote: > > core/src/main/scala/kafka/server/DelayedOperation.scala, lines 264-266 > > > > > > Not sure if we need this check. Since all writes to watchersForKey are > > sync-ed,

Re: Review Request 35201: Fix KAFKA-2253

2015-06-11 Thread Onur Karaman
> On June 11, 2015, 1:07 a.m., Jun Rao wrote: > > core/src/main/scala/kafka/server/DelayedOperation.scala, lines 264-266 > > > > > > Not sure if we need this check. Since all writes to watchersForKey are > > sync-ed,

RE: [DISCUSS] KIP-25 System test improvements

2015-06-11 Thread Aditya Auradkar
Hi Geoffrey, Thanks for the writeup. Couple of questions: - Is it possible to configure suites using ducktape? For example: assume all the tests in system_tests have been migrated to ducktape. Can I run a subset of all tests grouped by functional areas i.e. replication, broker failure etc. - Du

[jira] [Updated] (KAFKA-2136) Client side protocol changes to return quota delays

2015-06-11 Thread Aditya Auradkar (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aditya Auradkar updated KAFKA-2136: --- Labels: quotas (was: ) > Client side protocol changes to return quota delays > --

[jira] [Updated] (KAFKA-2171) System Test for Quotas

2015-06-11 Thread Aditya Auradkar (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aditya Auradkar updated KAFKA-2171: --- Labels: quotas (was: ) > System Test for Quotas > -- > >

[jira] [Updated] (KAFKA-2084) byte rate metrics per client ID (producer and consumer)

2015-06-11 Thread Aditya Auradkar (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aditya Auradkar updated KAFKA-2084: --- Labels: quotas (was: ) > byte rate metrics per client ID (producer and consumer) > --

[jira] [Updated] (KAFKA-2241) AbstractFetcherThread.shutdown() should not block on ReadableByteChannel.read(buffer)

2015-06-11 Thread Aditya Auradkar (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aditya Auradkar updated KAFKA-2241: --- Labels: quotas (was: ) > AbstractFetcherThread.shutdown() should not block on > ReadableByte

[jira] [Updated] (KAFKA-2191) Measured rate should not be infinite

2015-06-11 Thread Dong Lin (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Lin updated KAFKA-2191: Labels: quotas (was: ) > Measured rate should not be infinite > > >

[jira] [Updated] (KAFKA-2205) Generalize TopicConfigManager to handle multiple entity configs

2015-06-11 Thread Aditya Auradkar (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aditya Auradkar updated KAFKA-2205: --- Labels: quotas (was: ) > Generalize TopicConfigManager to handle multiple entity configs > --

RE: [DISCUSS] KIP-4 - Command line and centralized administrative operations (Thread 2)

2015-06-11 Thread Aditya Auradkar
I've made two changes to the document: - Removed the TMR evolution piece since we agreed to retain this. - Added two new API's to the admin client spec. (Alter and Describe config). Please review. Aditya From: Ashish Singh [asi...@cloudera.com] Sent: Frid

Re: Review Request 35201: Fix KAFKA-2253

2015-06-11 Thread Jun Rao
> On June 11, 2015, 1:07 a.m., Jun Rao wrote: > > core/src/main/scala/kafka/server/DelayedOperation.scala, lines 264-266 > > > > > > Not sure if we need this check. Since all writes to watchersForKey are > > sync-ed,

Re: Review Request 35231: Fix KAFKA-1740

2015-06-11 Thread Onur Karaman
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35231/#review87583 --- core/src/main/scala/kafka/coordinator/ConsumerCoordinator.scala

Re: [DISCUSS] KIP-25 System test improvements

2015-06-11 Thread Geoffrey Anderson
Hi Aditya, (1) There are a currently a few different ways to target a specific test or subset of tests. If for example tests were organized like the current system tests, where suites are grouped by directory, you could run cd ducktape replication_testsuite/ You can also target tests in a part

RE: [DISCUSS] KIP-4 - Command line and centralized administrative operations (Thread 2)

2015-06-11 Thread Aditya Auradkar
Andrii, Do we need a new voting thread for this KIP? The last round of votes had 3 binding +1's but there's been a fair amount of discussion since then. Aditya From: Aditya Auradkar Sent: Thursday, June 11, 2015 10:32 AM To: dev@kafka.apache.org Subject:

Re: [VOTE] KIP-25 System test improvements

2015-06-11 Thread Ewen Cheslack-Postava
+1 -Ewen On Thu, Jun 11, 2015 at 1:47 AM, Joe Stein wrote: > +1 > > ~ Joestein > On Jun 10, 2015 10:21 PM, "Neha Narkhede" wrote: > > > +1. Thanks Geoff! > > > > > > > > > > > > On Wed, Jun 10, 2015 at 6:20 PM -0700, "Gwen Shapira" < > > gshap...@cloudera.com> wrote: > > > > > > > > > > > > >

[jira] [Created] (KAFKA-2265) creating a topic with large number of partitions takes a long time

2015-06-11 Thread Jun Rao (JIRA)
Jun Rao created KAFKA-2265: -- Summary: creating a topic with large number of partitions takes a long time Key: KAFKA-2265 URL: https://issues.apache.org/jira/browse/KAFKA-2265 Project: Kafka Issue T

[jira] [Commented] (KAFKA-1367) Broker topic metadata not kept in sync with ZooKeeper

2015-06-11 Thread Joel Koshy (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14582433#comment-14582433 ] Joel Koshy commented on KAFKA-1367: --- [~singhashish] - yes that is a good summary. Broker

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations (Thread 2)

2015-06-11 Thread Joel Koshy
Discussion aside, was there any significant material change besides the additions below? If so, then we can avoid the overhead of another vote unless someone wants to down-vote these changes. Joel On Thu, Jun 11, 2015 at 06:36:36PM +, Aditya Auradkar wrote: > Andrii, > > Do we need a new vot

Re: Review Request 34789: Patch for KAFKA-2168

2015-06-11 Thread Jason Gustafson
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/34789/ --- (Updated June 11, 2015, 9:10 p.m.) Review request for kafka. Bugs: KAFKA-2168

[jira] [Updated] (KAFKA-2168) New consumer poll() can block other calls like position(), commit(), and close() indefinitely

2015-06-11 Thread Jason Gustafson (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Gustafson updated KAFKA-2168: --- Attachment: KAFKA-2168_2015-06-11_14:09:59.patch > New consumer poll() can block other calls l

[jira] [Commented] (KAFKA-2168) New consumer poll() can block other calls like position(), commit(), and close() indefinitely

2015-06-11 Thread Jason Gustafson (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14582524#comment-14582524 ] Jason Gustafson commented on KAFKA-2168: Updated reviewboard https://reviews.apach

[jira] [Commented] (KAFKA-2168) New consumer poll() can block other calls like position(), commit(), and close() indefinitely

2015-06-11 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14582528#comment-14582528 ] Jay Kreps commented on KAFKA-2168: -- Hey [~guozhang], have you had a chance to look at thi

Re: [DISCUSS] KIP-23 - Add JSON/CSV output and looping options to ConsumerGroupCommand

2015-06-11 Thread Ashish Singh
Hi Guys, This has been lying around for quite some time. Should I start a voting thread on this? On Thu, May 7, 2015 at 12:20 PM, Ashish Singh wrote: > Had to change the title of the page and that surprisingly changed the link > as well. KIP-23 is now available at here >

Pending review requests

2015-06-11 Thread Ashish Singh
Hey Guys, I have a few JIRAs in "patch available" state for some time. I will really appreciate if someone can review them. 1. https://issues.apache.org/jira/browse/KAFKA-2132 2. https://issues.apache.org/jira/browse/KAFKA-2005 3. https://issues.apache.org/jira/browse/KAFKA-1722 4. https://issues

[jira] [Commented] (KAFKA-1367) Broker topic metadata not kept in sync with ZooKeeper

2015-06-11 Thread Ashish K Singh (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14582532#comment-14582532 ] Ashish K Singh commented on KAFKA-1367: --- [~jjkoshy] thanks for confirming. I will ge

Re: Review Request 34789: Patch for KAFKA-2168

2015-06-11 Thread Jason Gustafson
> On June 9, 2015, 7:58 p.m., Jun Rao wrote: > > clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java, > > lines 797-798 > > > > > > Hmm, seekToBegining() is supposed to be a blocking call. Basic

Re: Review Request 34789: Patch for KAFKA-2168

2015-06-11 Thread Jason Gustafson
> On June 3, 2015, 6:44 a.m., Ewen Cheslack-Postava wrote: > > clients/src/main/java/org/apache/kafka/clients/consumer/internals/BrokerResponse.java, > > line 15 > > > > > > The classes named XResponse may be a bit con

Re: [DISCUSS] KIP-23 - Add JSON/CSV output and looping options to ConsumerGroupCommand

2015-06-11 Thread Gwen Shapira
Maybe bring it up at the next KIP call, to make sure everyone is aware? On Thu, Jun 11, 2015 at 2:17 PM, Ashish Singh wrote: > Hi Guys, > > This has been lying around for quite some time. Should I start a voting > thread on this? > > On Thu, May 7, 2015 at 12:20 PM, Ashish Singh wrote: > >> Had

[jira] [Updated] (KAFKA-2132) Move Log4J appender to a separate module

2015-06-11 Thread Gwen Shapira (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira updated KAFKA-2132: Summary: Move Log4J appender to a separate module (was: Move Log4J appender to clients module) > M

Re: [DISCUSS] KIP-23 - Add JSON/CSV output and looping options to ConsumerGroupCommand

2015-06-11 Thread Ashish Singh
Jun, Can we add this as part of next KIP's agenda? On Thu, Jun 11, 2015 at 3:00 PM, Gwen Shapira wrote: > Maybe bring it up at the next KIP call, to make sure everyone is aware? > > On Thu, Jun 11, 2015 at 2:17 PM, Ashish Singh wrote: > > Hi Guys, > > > > This has been lying around for quite s

Re: Review Request 33614: Patch for KAFKA-2132

2015-06-11 Thread Aditya Auradkar
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33614/#review87638 --- Hey Ashish, I've left a few minor comments. Thanks! log4j-appende

[jira] [Updated] (KAFKA-2005) Generate html report for system tests

2015-06-11 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao updated KAFKA-2005: --- Resolution: Fixed Fix Version/s: 0.8.3 Status: Resolved (was: Patch Available) Thanks for t

[jira] [Commented] (KAFKA-2005) Generate html report for system tests

2015-06-11 Thread Ashish K Singh (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14582646#comment-14582646 ] Ashish K Singh commented on KAFKA-2005: --- Thanks [~junrao] for reviewing and committi

[jira] [Updated] (KAFKA-2249) KafkaConfig does not preserve original Properties

2015-06-11 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao updated KAFKA-2249: --- Status: In Progress (was: Patch Available) > KafkaConfig does not preserve original Properties >

Re: Review Request 35347: Patch for KAFKA-2249

2015-06-11 Thread Jun Rao
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35347/#review87643 --- Thanks a lot for cleaning this up! A few comments below. core/src/

Re: Review Request 33620: Patch for KAFKA-1690

2015-06-11 Thread Ismael Juma
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33620/#review87646 --- clients/src/main/java/org/apache/kafka/common/network/SSLTransportL

Jenkins build is back to normal : Kafka-trunk #507

2015-06-11 Thread Apache Jenkins Server
See

[jira] [Commented] (KAFKA-2238) KafkaMetricsConfig cannot be configured in broker (KafkaConfig)

2015-06-11 Thread Aditya Auradkar (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14582679#comment-14582679 ] Aditya Auradkar commented on KAFKA-2238: [~junrao][~nehanarkhede] Can one of you r

[jira] [Commented] (KAFKA-2238) KafkaMetricsConfig cannot be configured in broker (KafkaConfig)

2015-06-11 Thread Gwen Shapira (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14582698#comment-14582698 ] Gwen Shapira commented on KAFKA-2238: - [~aauradkar] I made significant changes to Kafk

Re: Review Request 35347: Patch for KAFKA-2249

2015-06-11 Thread Gwen Shapira
> On June 11, 2015, 11:22 p.m., Jun Rao wrote: > > core/src/main/scala/kafka/server/KafkaConfig.scala, line 485 > > > > > > Is there a particular reason to change this to a long? It is used as LONG everywhere in the c

[jira] [Commented] (KAFKA-2238) KafkaMetricsConfig cannot be configured in broker (KafkaConfig)

2015-06-11 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14582722#comment-14582722 ] Jun Rao commented on KAFKA-2238: [~auradkar], took a look at your patch. Currently, the Ka

Re: Review Request 35261: Patch for KAFKA-2232

2015-06-11 Thread Jun Rao
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35261/#review87661 --- Thanks for the patch. A few comments below. clients/src/main/java/

Re: [VOTE] KIP-25 System test improvements

2015-06-11 Thread Jun Rao
+1 Thanks, Jun On Wed, Jun 10, 2015 at 6:10 PM, Geoffrey Anderson wrote: > Hi Kafka, > > After a few rounds of discussion on KIP-25, there doesn't seem to be > opposition, so I'd like to propose a vote. > > Thanks, > Geoff > > On Mon, Jun 8, 2015 at 10:56 PM, Geoffrey Anderson > wrote: > > >

Re: [VOTE] KIP-25 System test improvements

2015-06-11 Thread Jun Rao
+1 Thanks, Jun On Wed, Jun 10, 2015 at 6:10 PM, Geoffrey Anderson wrote: > Hi Kafka, > > After a few rounds of discussion on KIP-25, there doesn't seem to be > opposition, so I'd like to propose a vote. > > Thanks, > Geoff > > On Mon, Jun 8, 2015 at 10:56 PM, Geoffrey Anderson > wrote: > > >

[jira] [Resolved] (KAFKA-2263) Update "Is it possible to delete a topic" wiki FAQ answer

2015-06-11 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao resolved KAFKA-2263. Resolution: Fixed Thanks for reporting this. Updated the wiki. > Update "Is it possible to delete a topic"

Re: [Discussion] New Consumer API / Protocol

2015-06-11 Thread Jun Rao
Guozhang, Perhaps we can discuss this in our KIP hangout next week? Thanks, Jun On Tue, Jun 9, 2015 at 1:12 PM, Guozhang Wang wrote: > This email is to kick-off some discussion around the changes we want to > make on the new consumer APIs as well as their semantics. Here are a > not-comprehen

[jira] [Created] (KAFKA-2266) Client Selector can drop idle connections without notifying NetworkClient

2015-06-11 Thread Jason Gustafson (JIRA)
Jason Gustafson created KAFKA-2266: -- Summary: Client Selector can drop idle connections without notifying NetworkClient Key: KAFKA-2266 URL: https://issues.apache.org/jira/browse/KAFKA-2266 Project:

Review Request 35371: Patch for KAFKA-2266

2015-06-11 Thread Jason Gustafson
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35371/ --- Review request for kafka. Bugs: KAFKA-2266 https://issues.apache.org/jira/b

[jira] [Updated] (KAFKA-2266) Client Selector can drop idle connections without notifying NetworkClient

2015-06-11 Thread Jason Gustafson (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Gustafson updated KAFKA-2266: --- Attachment: KAFKA-2266.patch > Client Selector can drop idle connections without notifying Net

[jira] [Updated] (KAFKA-2266) Client Selector can drop idle connections without notifying NetworkClient

2015-06-11 Thread Jason Gustafson (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Gustafson updated KAFKA-2266: --- Status: Patch Available (was: Open) > Client Selector can drop idle connections without notif

[jira] [Commented] (KAFKA-2266) Client Selector can drop idle connections without notifying NetworkClient

2015-06-11 Thread Jason Gustafson (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14582762#comment-14582762 ] Jason Gustafson commented on KAFKA-2266: Created reviewboard https://reviews.apach

Re: Review Request 35371: Patch for KAFKA-2266

2015-06-11 Thread Jun Rao
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35371/#review87667 --- Thanks for the patch. Good catch! A minor comment below. clients/s

Re: Review Request 33614: Patch for KAFKA-2132

2015-06-11 Thread Jun Rao
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33614/#review87670 --- Thanks for the patch. A few comments below. build.gradle

[jira] [Updated] (KAFKA-2132) Move Log4J appender to a separate module

2015-06-11 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao updated KAFKA-2132: --- Status: In Progress (was: Patch Available) > Move Log4J appender to a separate module > -

Re: [DISCUSS] KIP-23 - Add JSON/CSV output and looping options to ConsumerGroupCommand

2015-06-11 Thread Neha Narkhede
Thanks for submitting the KIP, Ashish! Few questions. 1. Can you specify more details around how you expect csv output to be used. Same for json. 2. If we add these options, would you still need the old format. If csv/json offers more convenience, should we have a plan to phase out the old format?