Re: Multiple Processes Consuming from Same GroupID

I usually get this exception when I define > 2 partitions .. Current configuration : Single Topic - 4 partitions 1 Consumers Group - 10 Threads On Wed, Sep 11, 2013 at 10:24 PM, prashant amar wrote: > From the broker log: > > > INFO Reconnect due to socket error: (kafka.consumer.SimpleCons

Re: Multiple Processes Consuming from Same GroupID

>From the broker log: INFO Reconnect due to socket error: (kafka.consumer.SimpleConsumer) java.nio.channels.ClosedChannelException at kafka.network.BlockingChannel.send(BlockingChannel.scala:89) at kafka.consumer.SimpleConsumer.liftedTree1$1(SimpleConsumer.scala:72) at kafka.consumer.SimpleConsu

Re: Multiple Processes Consuming from Same GroupID

This means the broker somehow closed the socket connection. Anything in the broker log around the same time? Thanks, Jun On Wed, Sep 11, 2013 at 6:07 PM, prashant amar wrote: > Also noticed another issue > > Specified below is the current configuration > > Topic1 -> n Partitions -> 2 Consumer

[jira] [Commented] (KAFKA-1051) merge from 0.8 da4512174b6f7c395ffe053a86d2c6bb19d2538a to trunk

[ https://issues.apache.org/jira/browse/KAFKA-1051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13765141#comment-13765141 ] Jun Rao commented on KAFKA-1051: There's a problem in generating a reviewboard with this p

Re: Kafka patch review process

I think this is awesome! Two quick things: 1. I think once we have vetted review board and this tool it would be good to consolidate the committer workflow documentation. Currently we have a git workflow, a review board workflow and a review tool workflow. I actually do not have the ability to re

Re: Multiple Processes Consuming from Same GroupID

Also noticed another issue Specified below is the current configuration Topic1 -> n Partitions -> 2 Consumer Groups (gr1 and gr2) Topic2 -> n Partitions -> 2 Consumer Groups (gr1 and gr2) Notice that I have used the same naming convention on the consumer group set i.e. 'gr1' and 'gr2' are consum

Kafka patch review process

Hi, I'm proposing a new patch review process that will save time on code reviews and make the patch review process easier, with a one time setup cost. I wrote a tool that will do 2 things - 1. Create a patch and upload it to an existing JIRA 2. Create/update reviewboard with the same patch and l

Re: Multiple Processes Consuming from Same GroupID

I think you are hitting this - https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Whysomeoftheconsumersinaconsumergroupneverreceiveanymessage%3F Let us know if we can improve the documentation to make it clearer. Thanks, Neha On Wed, Sep 11, 2013 at 5:28 PM, prashant amar wrote: > Also

Re: Multiple Processes Consuming from Same GroupID

Also attempted another pattern where 1. Created a topic T with 'n' partitions. 2. Created a consumer group process with 'n + 1' threads subscribing from topic 'T' with a groupID 'y' 3. Added another consumer group process with 'n + 1' threads subscribing from same topic 'T' with same groupID 'z' (

Multiple Processes Consuming from Same GroupID

A Design Question that needs verification: 1. Created a topic T with 'n' partitions. 2. Created a consumer group process with 'n + 1' threads subscribing from topic 'T' with a groupID 'y' 3. Added another consumer group process with 'n + 1' threads subscribing from same topic 'T' with same groupID

Re: review board is available

Hooray! Thanks Neha. Guozhang On Wed, Sep 11, 2013 at 10:50 AM, Neha Narkhede wrote: > I wrote a wrapper script that will update the jira as well as the > reviewboard. So the proposed new patch review process will be - > > 1. Create JIRA > 2. Make code changes and commit to local branch > 3. Us

[jira] [Updated] (KAFKA-1051) merge from 0.8 da4512174b6f7c395ffe053a86d2c6bb19d2538a to trunk

[ https://issues.apache.org/jira/browse/KAFKA-1051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao updated KAFKA-1051: --- Status: Patch Available (was: Open) > merge from 0.8 da4512174b6f7c395ffe053a86d2c6bb19d2538a to trunk >

[jira] [Created] (KAFKA-1051) merge from 0.8 da4512174b6f7c395ffe053a86d2c6bb19d2538a to trunk

Jun Rao created KAFKA-1051: -- Summary: merge from 0.8 da4512174b6f7c395ffe053a86d2c6bb19d2538a to trunk Key: KAFKA-1051 URL: https://issues.apache.org/jira/browse/KAFKA-1051 Project: Kafka Issue Typ

Review Request 14091: Kafka patch review tool

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/14091/ --- Review request for kafka. Bugs: KAFKA-1053 https://issues.apache.org/jira/b

[jira] [Updated] (KAFKA-1003) ConsumerFetcherManager should pass clientId as metricsPrefix to AbstractFetcherManager

[ https://issues.apache.org/jira/browse/KAFKA-1003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede updated KAFKA-1003: - Attachment: (was: README.txt) > ConsumerFetcherManager should pass clientId as metricsPre

[jira] [Issue Comment Deleted] (KAFKA-1003) ConsumerFetcherManager should pass clientId as metricsPrefix to AbstractFetcherManager

[ https://issues.apache.org/jira/browse/KAFKA-1003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede updated KAFKA-1003: - Comment: was deleted (was: Command line interface client for JIRA This is a command line interfa

[jira] [Updated] (KAFKA-1003) ConsumerFetcherManager should pass clientId as metricsPrefix to AbstractFetcherManager

[ https://issues.apache.org/jira/browse/KAFKA-1003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede updated KAFKA-1003: - Attachment: README.txt > ConsumerFetcherManager should pass clientId as metricsPrefix to > A

[jira] [Commented] (KAFKA-1003) ConsumerFetcherManager should pass clientId as metricsPrefix to AbstractFetcherManager

[ https://issues.apache.org/jira/browse/KAFKA-1003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13764427#comment-13764427 ] Neha Narkhede commented on KAFKA-1003: -- Command line interface client for JIRA This

[jira] [Commented] (KAFKA-797) Cannot start Kafka 0.8 per Quick Start instructions

[ https://issues.apache.org/jira/browse/KAFKA-797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13764787#comment-13764787 ] Diederik van Liere commented on KAFKA-797: -- According to http://kafka.apache.org/0

[jira] [Created] (KAFKA-1053) Kafka patch review tool

Neha Narkhede created KAFKA-1053: Summary: Kafka patch review tool Key: KAFKA-1053 URL: https://issues.apache.org/jira/browse/KAFKA-1053 Project: Kafka Issue Type: New Feature Compo

[jira] [Commented] (KAFKA-1053) Kafka patch review tool

[ https://issues.apache.org/jira/browse/KAFKA-1053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13764743#comment-13764743 ] Neha Narkhede commented on KAFKA-1053: -- Updated reviewboard https://reviews.apache.or

[jira] [Commented] (KAFKA-1053) Kafka patch review tool

[ https://issues.apache.org/jira/browse/KAFKA-1053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13764741#comment-13764741 ] Neha Narkhede commented on KAFKA-1053: -- Thanks for the quick review Tejas 1. I belie

[jira] [Updated] (KAFKA-1053) Kafka patch review tool

[ https://issues.apache.org/jira/browse/KAFKA-1053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede updated KAFKA-1053: - Attachment: KAFKA-1053-v1.patch > Kafka patch review tool > --- > >

Re: Review Request 14091: Kafka patch review tool

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/14091/ --- (Updated Sept. 11, 2013, 8:44 p.m.) Review request for kafka. Bugs: KAFKA-105

[jira] [Updated] (KAFKA-1053) Kafka patch review tool

[ https://issues.apache.org/jira/browse/KAFKA-1053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede updated KAFKA-1053: - Attachment: KAFKA-1053-v2.patch > Kafka patch review tool > --- > >

Re: Review Request 14091: Kafka patch review tool

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/14091/ --- (Updated Sept. 11, 2013, 8:18 p.m.) Review request for kafka. Bugs: KAFKA-105

[jira] [Issue Comment Deleted] (KAFKA-1053) Kafka patch review tool

[ https://issues.apache.org/jira/browse/KAFKA-1053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede updated KAFKA-1053: - Comment: was deleted (was: Updated reviewboard) > Kafka patch review tool >

[jira] [Updated] (KAFKA-1053) Kafka patch review tool

[ https://issues.apache.org/jira/browse/KAFKA-1053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede updated KAFKA-1053: - Attachment: KAFKA-1053-v1.patch > Kafka patch review tool > --- > >

[jira] [Updated] (KAFKA-1053) Kafka patch review tool

[ https://issues.apache.org/jira/browse/KAFKA-1053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede updated KAFKA-1053: - Attachment: (was: KAFKA-1053-v1.patch) > Kafka patch review tool > --

[jira] [Commented] (KAFKA-1053) Kafka patch review tool

[ https://issues.apache.org/jira/browse/KAFKA-1053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13764712#comment-13764712 ] Neha Narkhede commented on KAFKA-1053: -- Updated reviewboard > Kafka

[jira] [Updated] (KAFKA-1053) Kafka patch review tool

[ https://issues.apache.org/jira/browse/KAFKA-1053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede updated KAFKA-1053: - Attachment: (was: KAFKA-1053-v1.patch) > Kafka patch review tool > --

[jira] [Updated] (KAFKA-1053) Kafka patch review tool

[ https://issues.apache.org/jira/browse/KAFKA-1053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede updated KAFKA-1053: - Attachment: (was: KAFKA-1053-v2.patch) > Kafka patch review tool > --

[jira] [Updated] (KAFKA-1053) Kafka patch review tool

[ https://issues.apache.org/jira/browse/KAFKA-1053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede updated KAFKA-1053: - Attachment: KAFKA-1053-v1.patch > Kafka patch review tool > --- > >

Re: Review Request 14091: Kafka patch review tool

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/14091/ --- (Updated Sept. 11, 2013, 7:56 p.m.) Review request for kafka. Bugs: KAFKA-105

Re: Review Request 14091: Kafka patch review tool

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/14091/ --- (Updated Sept. 11, 2013, 7:55 p.m.) Review request for kafka. Bugs: KAFKA-105

[jira] [Updated] (KAFKA-1053) Kafka patch review tool

[ https://issues.apache.org/jira/browse/KAFKA-1053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede updated KAFKA-1053: - Attachment: KAFKA-1053-v1.patch > Kafka patch review tool > --- > >

[jira] [Commented] (KAFKA-1053) Kafka patch review tool

[ https://issues.apache.org/jira/browse/KAFKA-1053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13764714#comment-13764714 ] Neha Narkhede commented on KAFKA-1053: -- Updated reviewboard https://reviews.apache.or

[jira] [Commented] (KAFKA-1053) Kafka patch review tool

[ https://issues.apache.org/jira/browse/KAFKA-1053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13764683#comment-13764683 ] Tejas Patil commented on KAFKA-1053: (1) In [0], "Setup" -> hyperlinks on steps 1 and

Review Request 14094: Kafka patch review tool

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/14094/ --- Review request for kafka. Bugs: KAFKA-1053 https://issues.apache.org/jira/b

Re: Review Request 14091: Kafka patch review tool

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/14091/ --- (Updated Sept. 11, 2013, 7:53 p.m.) Review request for kafka. Bugs: KAFKA-105

[jira] [Commented] (KAFKA-1051) merge from 0.8 da4512174b6f7c395ffe053a86d2c6bb19d2538a to trunk

[ https://issues.apache.org/jira/browse/KAFKA-1051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13764676#comment-13764676 ] Neha Narkhede commented on KAFKA-1051: -- This is a big change. Any chance you can give

Re: review board is available

Thanks for the review. I created a JIRA to track this - https://issues.apache.org/jira/browse/KAFKA-1053. Will be great if you can add your comments there. Thanks, Neha On Wed, Sep 11, 2013 at 11:28 AM, Tejas Patil wrote: > This is cool !!! I can't wait to use it. > > Some comments: > > (1) In

Re: review board is available

I wrote a wrapper script that will update the jira as well as the reviewboard. So the proposed new patch review process will be - 1. Create JIRA 2. Make code changes and commit to local branch 3. Use patch review tool

Re: review board is available

This is cool !!! I can't wait to use it. Some comments: (1) In [0], "Setup" -> hyperlinks on steps 1 and 2 loop to the same webpage. (2) I don't have much idea about the right place where the ".reviewboardrc" file should be, but it would be a good idea to commit it in the codebase like [1]. Also

[jira] [Commented] (KAFKA-1046) Added support for Scala 2.10 builds while maintaining compatibility with 2.8.x

[ https://issues.apache.org/jira/browse/KAFKA-1046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13764544#comment-13764544 ] Jun Rao commented on KAFKA-1046: Chris, I merged everything in 0.8 before your change to

[jira] [Updated] (KAFKA-1051) merge from 0.8 da4512174b6f7c395ffe053a86d2c6bb19d2538a to trunk

[ https://issues.apache.org/jira/browse/KAFKA-1051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao updated KAFKA-1051: --- Resolution: Fixed Status: Resolved (was: Patch Available) Since all merge conflicts are easy to reso

[jira] [Commented] (KAFKA-1051) merge from 0.8 da4512174b6f7c395ffe053a86d2c6bb19d2538a to trunk

[ https://issues.apache.org/jira/browse/KAFKA-1051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13764503#comment-13764503 ] Jun Rao commented on KAFKA-1051: Will leave this jira un-closed for a few days in case peo

[jira] [Commented] (KAFKA-970) ./sbt +package rebuilds the Hadoop consumer jar N times with the same output file

[ https://issues.apache.org/jira/browse/KAFKA-970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13764540#comment-13764540 ] Jun Rao commented on KAFKA-970: --- This affects the following jars. ./contrib/hadoop-consumer/

[jira] [Created] (KAFKA-1052) integrate add-partitions command into topicCommand

Jun Rao created KAFKA-1052: -- Summary: integrate add-partitions command into topicCommand Key: KAFKA-1052 URL: https://issues.apache.org/jira/browse/KAFKA-1052 Project: Kafka Issue Type: Bug

[jira] [Updated] (KAFKA-1051) merge from 0.8 da4512174b6f7c395ffe053a86d2c6bb19d2538a to trunk

[ https://issues.apache.org/jira/browse/KAFKA-1051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao updated KAFKA-1051: --- Attachment: kafka-1051.patch Attach a patch for review. Somehow, couldn't publish a reviewboard because of t

[jira] [Updated] (KAFKA-1003) ConsumerFetcherManager should pass clientId as metricsPrefix to AbstractFetcherManager

[ https://issues.apache.org/jira/browse/KAFKA-1003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede updated KAFKA-1003: - Attachment: (was: KAFKA-1003-v1.patch) > ConsumerFetcherManager should pass clientId as m

[jira] [Updated] (KAFKA-1003) ConsumerFetcherManager should pass clientId as metricsPrefix to AbstractFetcherManager

[ https://issues.apache.org/jira/browse/KAFKA-1003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede updated KAFKA-1003: - Attachment: KAFKA-1003-v1.patch > ConsumerFetcherManager should pass clientId as metricsPrefi

[jira] [Issue Comment Deleted] (KAFKA-1003) ConsumerFetcherManager should pass clientId as metricsPrefix to AbstractFetcherManager

[ https://issues.apache.org/jira/browse/KAFKA-1003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede updated KAFKA-1003: - Comment: was deleted (was: testing the command line script) > ConsumerFetcherManager should

[jira] [Commented] (KAFKA-1003) ConsumerFetcherManager should pass clientId as metricsPrefix to AbstractFetcherManager

[ https://issues.apache.org/jira/browse/KAFKA-1003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13764426#comment-13764426 ] Neha Narkhede commented on KAFKA-1003: -- testing the command line script