[jira] [Created] (KAFKA-2382) KafkaConsumer seek methods should throw an exception when called for partitions not assigned to this consumer instance

Ewen Cheslack-Postava created KAFKA-2382: Summary: KafkaConsumer seek methods should throw an exception when called for partitions not assigned to this consumer instance Key: KAFKA-2382 URL: https://issues

[jira] [Updated] (KAFKA-2350) Add KafkaConsumer pause capability

[ https://issues.apache.org/jira/browse/KAFKA-2350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ewen Cheslack-Postava updated KAFKA-2350: - Component/s: consumer > Add KafkaConsumer pause capability > -

[jira] [Updated] (KAFKA-2343) Clarify KafkaConsumer.poll rebalance behavior

[ https://issues.apache.org/jira/browse/KAFKA-2343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ewen Cheslack-Postava updated KAFKA-2343: - Component/s: consumer > Clarify KafkaConsumer.poll rebalance behavior > --

Re: Number of kafka topics/partitions supported per cluster of n nodes

I think these would definitely be useful statistics to have and I've tried to do similar tests! The biggest difference is probably going to be the hardware specs on whatever cluster you decide to run it on. Maybe benchmarks performed on different AWS servers would be helpful too, but I'd like to se

Re: Kafka Consumer thoughts

On Tue, Jul 28, 2015 at 5:18 PM, Guozhang Wang wrote: > I think Ewen has proposed these APIs for using callbacks along with > returning future in the commit calls, i.e. something similar to: > > public Future commit(ConsumerCommitCallback callback); > > public Future commit(Map offsets, > Consume

[jira] [Commented] (KAFKA-2255) Missing documentation for max.in.flight.requests.per.connection

[ https://issues.apache.org/jira/browse/KAFKA-2255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14645414#comment-14645414 ] Navina Ramesh commented on KAFKA-2255: -- Thanks! :) > Missing documentation for max.i

[jira] [Commented] (KAFKA-2350) Add KafkaConsumer pause capability

[ https://issues.apache.org/jira/browse/KAFKA-2350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14645343#comment-14645343 ] ASF GitHub Bot commented on KAFKA-2350: --- GitHub user hachikuji opened a pull request

[GitHub] kafka pull request: KAFKA-2350; KafkaConsumer pause/resume API

GitHub user hachikuji opened a pull request: https://github.com/apache/kafka/pull/100 KAFKA-2350; KafkaConsumer pause/resume API You can merge this pull request into a Git repository by running: $ git pull https://github.com/hachikuji/kafka KAFKA-2350 Alternatively you can re

Re: Review Request 33620: Patch for KAFKA-1690

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33620/#review93332 --- clients/src/main/java/org/apache/kafka/common/network/PlainTextTran

Re: Build failed in Jenkins: KafkaPreCommit #167

Anyone knows how this "Could not open buildscript class" issue could happen and can we fix it on our side or it is a general jenkins issue? Guozhang On Tue, Jul 28, 2015 at 5:39 PM, Apache Jenkins Server < jenk...@builds.apache.org> wrote: > See

Build failed in Jenkins: KafkaPreCommit #167

See Changes: [wangguoz] KAFKA-2276; KIP-25 initial patch -- Started by an SCM change Building remotely on ubuntu3 (Ubuntu ubuntu legacy-ubuntu) in workspace

[jira] [Commented] (KAFKA-2276) Initial patch for KIP-25

[ https://issues.apache.org/jira/browse/KAFKA-2276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14645268#comment-14645268 ] Guozhang Wang commented on KAFKA-2276: -- Thanks for the patch [~granders], it is commi

[jira] [Updated] (KAFKA-2276) Initial patch for KIP-25

[ https://issues.apache.org/jira/browse/KAFKA-2276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-2276: - Resolution: Fixed Fix Version/s: (was: 0.8.3) 0.9.0 Status

[jira] [Commented] (KAFKA-2276) Initial patch for KIP-25

[ https://issues.apache.org/jira/browse/KAFKA-2276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14645265#comment-14645265 ] ASF GitHub Bot commented on KAFKA-2276: --- Github user asfgit closed the pull request

[GitHub] kafka pull request: KAFKA-2276: KIP-25 initial patch

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/70 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enable

Re: Kafka Consumer thoughts

I think Ewen has proposed these APIs for using callbacks along with returning future in the commit calls, i.e. something similar to: public Future commit(ConsumerCommitCallback callback); public Future commit(Map offsets, ConsumerCommitCallback callback); At that time I was slightly intending no

Jenkins build is back to normal : KafkaPreCommit #166

See

[jira] [Resolved] (KAFKA-2089) MetadataTest transient failure

[ https://issues.apache.org/jira/browse/KAFKA-2089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-2089. -- Resolution: Fixed Fix Version/s: 0.8.3 > MetadataTest transient failure > ---

[jira] [Commented] (KAFKA-2089) MetadataTest transient failure

[ https://issues.apache.org/jira/browse/KAFKA-2089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14645192#comment-14645192 ] Guozhang Wang commented on KAFKA-2089: -- [~rsivaram] Thanks for the patch, just review

Re: Review Request 33125: Add comment to timing fix

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33125/#review93368 --- Ship it! Ship It! - Guozhang Wang On April 13, 2015, 7:15 p.m.,

Re: Review Request 36681: Patch for KAFKA-2275

> On July 28, 2015, 10:19 p.m., Guozhang Wang wrote: > > clients/src/test/java/org/apache/kafka/clients/consumer/internals/FetcherTest.java, > > line 42 > > > > > > HashMap is no longer used anywhere, and running che

[jira] [Commented] (KAFKA-2255) Missing documentation for max.in.flight.requests.per.connection

[ https://issues.apache.org/jira/browse/KAFKA-2255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14645166#comment-14645166 ] Aditya Auradkar commented on KAFKA-2255: This was brought up during a mailing list

[jira] [Assigned] (KAFKA-2255) Missing documentation for max.in.flight.requests.per.connection

[ https://issues.apache.org/jira/browse/KAFKA-2255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aditya Auradkar reassigned KAFKA-2255: -- Assignee: Aditya Auradkar > Missing documentation for max.in.flight.requests.per.connec

Re: Kafka Consumer thoughts

Hey Adi, When we designed the initial version, the producer API was still changing. I thought about adding the Future and then just didn't get to it. I agree that we should look into adding it for consistency. Thanks, Neha On Tue, Jul 28, 2015 at 1:51 PM, Aditya Auradkar wrote: > Great discuss

[jira] [Updated] (KAFKA-2275) Add a ListTopics() API to the new consumer

[ https://issues.apache.org/jira/browse/KAFKA-2275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-2275: - Resolution: Fixed Status: Resolved (was: Patch Available) Thanks for the patch, committed

Re: [DISCUSS] KIP-28 - Add a transform client for data processing

The second question that came up on the KIP was how do joins and aggregations work. A lot of implicit thinking went into Kafka's data model to support stream processing so there is an idea of how this should work but it isn't exactly obvious. Let me go through the idea of how a processor is meant t

Re: Review Request 36681: Patch for KAFKA-2275

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/36681/#review93361 --- Ship it! clients/src/test/java/org/apache/kafka/clients/consumer/i

Re: [DISCUSS] KIP-28 - Add a transform client for data processing

Thanks Guazhang! Much clearer now, at least for me. Few comments / questions: 1. Perhaps punctuate(int numRecords) will be a nice API addition, some use-cases have record-count based windows, rather than time-based.. 2. The diagram for "Flexible partition distribution" shows two joins. Is the ide

Re: Build failed in Jenkins: KafkaPreCommit #165

I tried the failing test locally and it passed. On Tue, Jul 28, 2015 at 2:59 PM, Apache Jenkins Server < jenk...@builds.apache.org> wrote: > See > > Changes: > > [wangguoz] KAFKA-2301: Warn ConsumerOffsetChecker as deprecated; reviewed >

Build failed in Jenkins: KafkaPreCommit #165

See Changes: [wangguoz] KAFKA-2301: Warn ConsumerOffsetChecker as deprecated; reviewed by Ewen Cheslack-Postava and Guozhang Wang [wangguoz] KAFKA-2381: Fix concurrent modification on assigned partition while looping over it; reviewed

Re: New Producer and "acks" configuration

Would also be nice if max inflights requests was documented :) https://issues.apache.org/jira/browse/KAFKA-2255 This is one of those things that would be nice to mention in docs On Mon, Jul 27, 2015 at 10:57 AM, Ewen Cheslack-Postava wrote: > If only we had some sort of system test framework

Re: Review Request 36871: Patch for KAFKA-2381

> On July 28, 2015, 8:07 p.m., Edward Ribeiro wrote: > > clients/src/main/java/org/apache/kafka/clients/consumer/internals/SubscriptionState.java, > > line 87 > > > > > > Sorry for being late to the party (I have bee

Re: New Producer and "acks" configuration

But on the server side, the code path for acks = 0 and acks = 1 is the same... its up to the client to implement acks = 0 and acks = 1 correctly (the way our java client fakes a response for acks = 0). So its not really a server feature, more of a protocol feature that needs to be implemented corre

[jira] [Updated] (KAFKA-2329) Consumers balance fails when multiple consumers are started simultaneously.

[ https://issues.apache.org/jira/browse/KAFKA-2329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-2329: - Status: In Progress (was: Patch Available) > Consumers balance fails when multiple consumers are

[jira] [Updated] (KAFKA-1782) Junit3 Misusage

[ https://issues.apache.org/jira/browse/KAFKA-1782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-1782: - Status: In Progress (was: Patch Available) > Junit3 Misusage > --- > >

[jira] [Updated] (KAFKA-2347) Add setConsumerRebalanceListener method to ZookeeperConsuemrConnector java api.

[ https://issues.apache.org/jira/browse/KAFKA-2347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-2347: - Resolution: Fixed Fix Version/s: 0.8.3 Status: Resolved (was: Patch Available)

Re: Review Request 36593: Patch for KAFKA-2347

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/36593/#review93352 --- Ship it! Ship It! - Guozhang Wang On July 18, 2015, 10:14 p.m.,

[jira] [Updated] (KAFKA-2381) Possible ConcurrentModificationException while unsubscribing from a topic in new consumer

[ https://issues.apache.org/jira/browse/KAFKA-2381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-2381: - Resolution: Fixed Fix Version/s: 0.8.3 Status: Resolved (was: Patch Available)

[jira] [Commented] (KAFKA-2301) Deprecate ConsumerOffsetChecker

[ https://issues.apache.org/jira/browse/KAFKA-2301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14645072#comment-14645072 ] Guozhang Wang commented on KAFKA-2301: -- Committed to trunk, thanks. > Deprecate Cons

Re: Review Request 36871: Patch for KAFKA-2381

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/36871/#review93351 --- Ship it! Ship It! - Guozhang Wang On July 28, 2015, 4:17 p.m., A

Re: Review Request 36871: Patch for KAFKA-2381

> On July 28, 2015, 8:07 p.m., Edward Ribeiro wrote: > > clients/src/main/java/org/apache/kafka/clients/consumer/internals/SubscriptionState.java, > > line 87 > > > > > > Sorry for being late to the party (I have bee

Re: Review Request 36871: Patch for KAFKA-2381

> On July 28, 2015, 8:07 p.m., Edward Ribeiro wrote: > > clients/src/main/java/org/apache/kafka/clients/consumer/internals/SubscriptionState.java, > > line 87 > > > > > > Sorry for being late to the party (I have bee

[jira] [Updated] (KAFKA-2301) Deprecate ConsumerOffsetChecker

[ https://issues.apache.org/jira/browse/KAFKA-2301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-2301: - Resolution: Fixed Status: Resolved (was: Patch Available) > Deprecate ConsumerOffsetCheck

Re: Review Request 35850: Patch for KAFKA-2301

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35850/#review93346 --- Ship it! Ship It! - Guozhang Wang On July 2, 2015, 4:04 p.m., As

Re: Kafka Consumer thoughts

Great discussion everyone! One general comment on the sync/async API's on the new consumer. I think the producer tackles sync vs async API's well. For API's that can either be sync or async, can we simply return a future? That seems more elegant for the API's that make sense either in both flavors

Re: error while high level consumer

Thanks Mayuresh and Jiangjie for your valuable insights. Bouncing took care of it. Thanks, Kris On Mon, Jul 27, 2015 at 4:06 PM, Mayuresh Gharat wrote: > Try bouncing the host that appears in the stored data section. > > Thanks, > > Mayuresh > > On Mon, Jul 27, 2015 at 3:41 PM, Jiangjie Qin >

Re: Review Request 36871: Patch for KAFKA-2381

> On July 28, 2015, 8:07 p.m., Edward Ribeiro wrote: > > clients/src/main/java/org/apache/kafka/clients/consumer/internals/SubscriptionState.java, > > line 87 > > > > > > Sorry for being late to the party (I have bee

Re: Review Request 36871: Patch for KAFKA-2381

> On July 28, 2015, 8:07 p.m., Edward Ribeiro wrote: > > clients/src/main/java/org/apache/kafka/clients/consumer/internals/SubscriptionState.java, > > line 87 > > > > > > Sorry for being late to the party (I have bee

Re: Review Request 36871: Patch for KAFKA-2381

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/36871/#review9 --- clients/src/main/java/org/apache/kafka/clients/consumer/internals/S

[jira] [Commented] (KAFKA-2145) An option to add topic owners.

[ https://issues.apache.org/jira/browse/KAFKA-2145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14644941#comment-14644941 ] Guozhang Wang commented on KAFKA-2145: -- [~junrao] seems you have already started revi

[jira] [Updated] (KAFKA-2145) An option to add topic owners.

[ https://issues.apache.org/jira/browse/KAFKA-2145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-2145: - Reviewer: Jun Rao > An option to add topic owners. > --- > >

[jira] [Commented] (KAFKA-2229) Phase 1: Requests and KafkaApis

[ https://issues.apache.org/jira/browse/KAFKA-2229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14644938#comment-14644938 ] Guozhang Wang commented on KAFKA-2229: -- [~junrao] could you help continue reviewing i

[jira] [Commented] (KAFKA-2351) Brokers are having a problem shutting down correctly

[ https://issues.apache.org/jira/browse/KAFKA-2351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14644934#comment-14644934 ] Guozhang Wang commented on KAFKA-2351: -- [~mgharat] Could you change the status to "Pa

[jira] [Updated] (KAFKA-2301) Deprecate ConsumerOffsetChecker

[ https://issues.apache.org/jira/browse/KAFKA-2301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-2301: - Reviewer: Guozhang Wang (was: Jun Rao) > Deprecate ConsumerOffsetChecker > --

[jira] [Updated] (KAFKA-2275) Add a ListTopics() API to the new consumer

[ https://issues.apache.org/jira/browse/KAFKA-2275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-2275: - Reviewer: Guozhang Wang > Add a ListTopics() API to the new consumer > ---

[jira] [Updated] (KAFKA-2351) Brokers are having a problem shutting down correctly

[ https://issues.apache.org/jira/browse/KAFKA-2351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-2351: - Reviewer: Jun Rao > Brokers are having a problem shutting down correctly > ---

[jira] [Updated] (KAFKA-2347) Add setConsumerRebalanceListener method to ZookeeperConsuemrConnector java api.

[ https://issues.apache.org/jira/browse/KAFKA-2347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-2347: - Reviewer: Guozhang Wang > Add setConsumerRebalanceListener method to ZookeeperConsuemrConnector ja

[jira] [Updated] (KAFKA-2381) Possible ConcurrentModificationException while unsubscribing from a topic in new consumer

[ https://issues.apache.org/jira/browse/KAFKA-2381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-2381: - Reviewer: Guozhang Wang > Possible ConcurrentModificationException while unsubscribing from a topi

[jira] [Updated] (KAFKA-1683) Implement a "session" concept in the socket server

[ https://issues.apache.org/jira/browse/KAFKA-1683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-1683: - Status: In Progress (was: Patch Available) > Implement a "session" concept in the socket server >

[jira] [Updated] (KAFKA-2100) Client Error doesn't preserve or display original server error code when it is an unknown code

[ https://issues.apache.org/jira/browse/KAFKA-2100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-2100: - Reviewer: Gwen Shapira > Client Error doesn't preserve or display original server error code when

[jira] [Updated] (KAFKA-2229) Phase 1: Requests and KafkaApis

[ https://issues.apache.org/jira/browse/KAFKA-2229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-2229: - Reviewer: Jun Rao > Phase 1: Requests and KafkaApis > --- > >

[jira] [Updated] (KAFKA-1695) Authenticate connection to Zookeeper

[ https://issues.apache.org/jira/browse/KAFKA-1695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-1695: - Reviewer: Gwen Shapira > Authenticate connection to Zookeeper > --

[jira] [Updated] (KAFKA-2145) An option to add topic owners.

[ https://issues.apache.org/jira/browse/KAFKA-2145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Parth Brahmbhatt updated KAFKA-2145: Status: Patch Available (was: In Progress) > An option to add topic owners. >

[jira] [Updated] (KAFKA-1695) Authenticate connection to Zookeeper

[ https://issues.apache.org/jira/browse/KAFKA-1695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Parth Brahmbhatt updated KAFKA-1695: Status: Patch Available (was: In Progress) > Authenticate connection to Zookeeper > ---

[jira] [Updated] (KAFKA-313) Add JSON/CSV output and looping options to ConsumerGroupCommand

[ https://issues.apache.org/jira/browse/KAFKA-313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-313: Reviewer: Gwen Shapira (was: Joel Koshy) > Add JSON/CSV output and looping options to ConsumerGroupC

Re: [DISCUSS] KIP-28 - Add a transform client for data processing

Here is the link to the original prototype we started with. I wouldn't focus to heavily on the details of this code or the api, but I think it gives the an idea of the lowest level api, amount of code, etc. It was basically a clone of Samza built on Kafka using the new consumer protocol just to exp

[jira] [Commented] (KAFKA-1683) Implement a "session" concept in the socket server

[ https://issues.apache.org/jira/browse/KAFKA-1683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14644840#comment-14644840 ] Ismael Juma commented on KAFKA-1683: OK, it depends on it. Understood, thanks. > Impl

[jira] [Commented] (KAFKA-1683) Implement a "session" concept in the socket server

[ https://issues.apache.org/jira/browse/KAFKA-1683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14644830#comment-14644830 ] Parth Brahmbhatt commented on KAFKA-1683: - I did not include this as part of Autho

Re: New Producer and "acks" configuration

For the new java producer this doesn't result in a big perf difference (at least in my testing) so there isn't a good reason to use acks=0. However this is a protocol and server feature, not a client feature. For the scala client and many of the other blocking clients the perf difference is quite s

[jira] [Commented] (KAFKA-1683) Implement a "session" concept in the socket server

[ https://issues.apache.org/jira/browse/KAFKA-1683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14644823#comment-14644823 ] Ismael Juma commented on KAFKA-1683: So merging the Authorizer will effectively includ

[jira] [Commented] (KAFKA-1683) Implement a "session" concept in the socket server

[ https://issues.apache.org/jira/browse/KAFKA-1683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14644805#comment-14644805 ] Parth Brahmbhatt commented on KAFKA-1683: - FYI, I relied on this Session object as

Re: KIP Meeting starting now

Hey Guozhang, Can you forward the invite again? Thanks, Mayuresh On Tue, Jul 28, 2015 at 11:04 AM, Guozhang Wang wrote: > Hi, > > Just a reminder that today's KIP meeting is on GoToMeeting, not on Hangout. > Please find the link on the invite to join. > > -- Guozhang > -- -Regards, Mayure

KIP Meeting starting now

Hi, Just a reminder that today's KIP meeting is on GoToMeeting, not on Hangout. Please find the link on the invite to join. -- Guozhang

Re: Review Request 36871: Patch for KAFKA-2381

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/36871/#review93321 --- Ship it! Ship It! - Jason Gustafson On July 28, 2015, 4:17 p.m.,

KAfka Mirror Maker

Hi, I'm using Mirror Maker with a cluster of 3 nodes and cluster of 5 nodes. I would like to ask - is the number of nodes a restriction for Mirror Maker? Also, are there any other restrictions or properties that should be common across both the clusters so that they continue mirroring. I'm aski

[jira] [Commented] (KAFKA-2100) Client Error doesn't preserve or display original server error code when it is an unknown code

[ https://issues.apache.org/jira/browse/KAFKA-2100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14644699#comment-14644699 ] David Jacot commented on KAFKA-2100: [~gwenshap] Could you review this one when you ha

Re: New Producer and "acks" configuration

I seems to me that performance wide, acks=1 and acks=0 will be pretty much the same if max.in.flight.request.per.connection is set to very high and does not cause a request sending to block. In new producer, if there are send failure from time to time, I would guess asks=1 would even have better p

Re: Review Request 34492: Patch for KAFKA-2210

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/34492/#review93302 --- I did an initial pass and left some comments. Some of them are quest

Re: Number of kafka topics/partitions supported per cluster of n nodes

I would be using the servers available at my place of work. I dont have access to AWS servers. I would starting off with a small number of nodes in the cluster and then plot a graph with x-axis as the number of servers in the cluster and y-axis as the number of topics with partitions, before the cl

Re: Number of kafka topics/partitions supported per cluster of n nodes

@Jiefu Gong, Are the results of your tests available publicly? Regards, Prabhjot On Tue, Jul 28, 2015 at 10:35 PM, Prabhjot Bharaj wrote: > I would be using the servers available at my place of work. I dont have > access to AWS servers. I would starting off with a small number of nodes in > th

[jira] [Commented] (KAFKA-2347) Add setConsumerRebalanceListener method to ZookeeperConsuemrConnector java api.

[ https://issues.apache.org/jira/browse/KAFKA-2347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14644656#comment-14644656 ] Ashish K Singh commented on KAFKA-2347: --- [~ijuma] thanks for the info. > Add setCon

[jira] [Commented] (KAFKA-2347) Add setConsumerRebalanceListener method to ZookeeperConsuemrConnector java api.

[ https://issues.apache.org/jira/browse/KAFKA-2347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14644624#comment-14644624 ] Ismael Juma commented on KAFKA-2347: [~junrao] is away for this week. Maybe [~guozhang

Re: Review Request 36871: Patch for KAFKA-2381

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/36871/#review93306 --- Ship it! Code change looks good. Not very familiar with the tests,

[jira] [Commented] (KAFKA-2347) Add setConsumerRebalanceListener method to ZookeeperConsuemrConnector java api.

[ https://issues.apache.org/jira/browse/KAFKA-2347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14644606#comment-14644606 ] Ashish K Singh commented on KAFKA-2347: --- This has been +1'd by multiple reviewers. [

[jira] [Commented] (KAFKA-2381) Possible ConcurrentModificationException while unsubscribing from a topic in new consumer

[ https://issues.apache.org/jira/browse/KAFKA-2381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14644595#comment-14644595 ] Ashish K Singh commented on KAFKA-2381: --- Updated reviewboard https://reviews.apache.

Re: Review Request 36871: Patch for KAFKA-2381

> On July 28, 2015, 8:16 a.m., Ismael Juma wrote: > > clients/src/main/java/org/apache/kafka/clients/consumer/internals/SubscriptionState.java, > > line 86 > > > > > > A minor optimisation is to use `ArrayList` here

[jira] [Updated] (KAFKA-2381) Possible ConcurrentModificationException while unsubscribing from a topic in new consumer

[ https://issues.apache.org/jira/browse/KAFKA-2381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish K Singh updated KAFKA-2381: -- Attachment: KAFKA-2381_2015-07-28_09:17:35.patch > Possible ConcurrentModificationException whil

Re: Review Request 36871: Patch for KAFKA-2381

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/36871/ --- (Updated July 28, 2015, 4:17 p.m.) Review request for kafka. Bugs: KAFKA-2381

[jira] [Commented] (KAFKA-1683) Implement a "session" concept in the socket server

[ https://issues.apache.org/jira/browse/KAFKA-1683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14644396#comment-14644396 ] Ismael Juma commented on KAFKA-1683: [~gwenshap], what is the status of this? Is it so

Re: Number of kafka topics/partitions supported per cluster of n nodes

Sure. I would be doing that. I have seen that if I have 5-7 topics with 256 partitions each on a machine with 4CPUs, 8GB RAM, the jvm crashes with OutOfMemoryError And, this happens in many machines in the cluster. (I'll update the exact number as well) I was wondering how I could tune the JVM to

Re: [DISCUSS] Partitioning in Kafka

Jason, Thanks for starting the discussion and for your very concise (and correct) summary. Ewen, while what you say is true, those kinds of detasets (large number of keys with skew) are very typical in the Web (think Twitter users, or Web pages, or even just plain text). If you want to compute an

Re: [DISCUSS] KIP-27 - Conditional Publish

Jiangjie: I think giving users the possibility of defining a custom policy for handling rejections is a good idea. For instance, this will allow Kafka to act as an event store in an Event Sourcing application. If the event(s) are rejected by the store, the original command may need to be re-validat

Re: Review Request 36871: Patch for KAFKA-2381

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/36871/#review93256 --- clients/src/main/java/org/apache/kafka/clients/consumer/internals/S

Re: [DISCUSS] KIP-28 - Add a transform client for data processing

I have updated the wiki page incorporating people's comments, please feel free to take another look before today's meeting. On Mon, Jul 27, 2015 at 11:19 PM, Yi Pan wrote: > Hi, Jay, > > {quote} > 1. Yeah we are going to try to generalize the partition management stuff. > We'll get a wiki/JIRA u