Re: [Kafka MirrorMaker] Message with Custom Partition Logic

Hi Neha and Guozhang, As long as stickiness is maintain consistently to a particular partition in target DC that is great so we can do per DC and across DC aggregation. How about non hash based instead of range based partitioning ? eg Key start with "a" then send message to partition 1 to 10, i

Re: [Kafka MirrorMaker] Message with Custom Partition Logic

Bhavesh, As Neha said, with more partitions on the destination brokers, events that are belong to the same partition in the source cluster may be distributed to different partitions in the destination cluster. Guozhang On Mon, Aug 11, 2014 at 9:35 PM, Neha Narkhede wrote: > Bhavesh, > > For k

[jira] [Commented] (KAFKA-1587) Possible Memory Leak when we use Kafka 8.0 Producer for sending messages

[ https://issues.apache.org/jira/browse/KAFKA-1587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14093734#comment-14093734 ] Neha Narkhede commented on KAFKA-1587: -- It looks like you know you are running a memo

[jira] [Resolved] (KAFKA-1584) TestUtils createBrokerConfig method argument mismatch

[ https://issues.apache.org/jira/browse/KAFKA-1584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede resolved KAFKA-1584. -- Resolution: Won't Fix Please direct such questions to the mailing list instead of creating JIRA

[jira] [Closed] (KAFKA-1584) TestUtils createBrokerConfig method argument mismatch

[ https://issues.apache.org/jira/browse/KAFKA-1584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede closed KAFKA-1584. > TestUtils createBrokerConfig method argument mismatch > -

Re: [Kafka MirrorMaker] Message with Custom Partition Logic

Bhavesh, For keyed data, the mirror maker will just distribute data based on hash(key)%num_partitions. If num_partitions is different in the target DC (which it is), a message that lived in partition 0 in the source cluster might end up in partition 10 in the target cluster. Thanks, Neha On Mon

Re: Stable Version of New Producer on Trunk

Bhavesh, It is best to build from trunk since it has the latest bug fixes that some of us found at LinkedIn. Let us know how your upgrade goes. Thanks Neha On Mon, Aug 11, 2014 at 7:17 PM, Bhavesh Mistry wrote: > Hi Guozhang, > > Yes, I understand this but, we need to develop and I just need

[jira] [Assigned] (KAFKA-1589) Strengthen System Tests

[ https://issues.apache.org/jira/browse/KAFKA-1589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira reassigned KAFKA-1589: --- Assignee: Gwen Shapira > Strengthen System Tests > --- > >

[jira] [Updated] (KAFKA-1589) Strengthen System Tests

[ https://issues.apache.org/jira/browse/KAFKA-1589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede updated KAFKA-1589: - Labels: newbie (was: ) > Strengthen System Tests > --- > > K

[jira] [Commented] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names

[ https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14093709#comment-14093709 ] Jun Rao commented on KAFKA-1481: I am not sure exactly what your use cases are, but I was

[jira] [Commented] (KAFKA-1589) Strengthen System Tests

[ https://issues.apache.org/jira/browse/KAFKA-1589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14093658#comment-14093658 ] Joe Stein commented on KAFKA-1589: -- This would be a great contribution. I would also ask

Re: [Kafka MirrorMaker] Message with Custom Partition Logic

Hi Guozhang, We are using Kafka 0.8.1 for all producer consumer and MM. We have 32 partition in source (local) per DC and we have 100 in target (Central) DC. Is there any configuration on MM for this etc ? Thanks, Bhavesh On Mon, Aug 11, 2014 at 4:33 PM, Guozhang Wang wrote: > Hi Bhavesh,

Re: Stable Version of New Producer on Trunk

Hi Guozhang, Yes, I understand this but, we need to develop and I just need a tag or branch name that is stable that is running in production environment at Linked-In or any other production environment. We probably can not wait for 0.8.2 official release. Or do you recommend building from maste

[jira] [Commented] (KAFKA-1477) add authentication layer and initial JKS x509 implementation for brokers, producers and consumer for network communication

[ https://issues.apache.org/jira/browse/KAFKA-1477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14093536#comment-14093536 ] Albert Strasheim commented on KAFKA-1477: - CloudFlare is also very interested in t

[jira] [Updated] (KAFKA-1584) TestUtils createBrokerConfig method argument mismatch

[ https://issues.apache.org/jira/browse/KAFKA-1584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] parin jogani updated KAFKA-1584: Description: Trying to write a unit test case for Kafka, and stuck with strange createBrokerConfig

[jira] [Commented] (KAFKA-1587) Possible Memory Leak when we use Kafka 8.0 Producer for sending messages

[ https://issues.apache.org/jira/browse/KAFKA-1587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14093519#comment-14093519 ] Gopinath Sundaram commented on KAFKA-1587: -- HI Kafka Team, Any update on this? H

Re: Review Request 24006: Patch for KAFKA-1420

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/24006/#review50264 --- Looks good to me. Can other committers double-check it? - Guozhang

Re: Stable Version of New Producer on Trunk

The new producer has not been officially released yet, we plan to include it in 0.8.2. Guozhang On Mon, Aug 11, 2014 at 1:29 PM, Bhavesh Mistry wrote: > Hi Kafka Dev, > > We are paining to upgrade to Java Producer (with Callback) on trunk. So > questions is which tag or branch we can use for

Re: [Kafka MirrorMaker] Message with Custom Partition Logic

Hi Bhavesh, What is the number of partitions on the source and target clusters, and what version of Kafka MM are you using? Guozhang On Mon, Aug 11, 2014 at 1:21 PM, Bhavesh Mistry wrote: > HI Kafka Dev Team, > > > > We have to aggregate events (count) per DC and across DCs for one of topic.

Re: Review Request 24196: Patch for KAFKA-1567

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/24196/ --- (Updated Aug. 11, 2014, 10:55 p.m.) Review request for kafka. Bugs: KAFKA-156

[jira] [Updated] (KAFKA-1567) Metric memory leaking after closing the clients

[ https://issues.apache.org/jira/browse/KAFKA-1567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiangjie Qin updated KAFKA-1567: Attachment: KAFKA-1567_2014-08-11_15:55:26.patch > Metric memory leaking after closing the clients

[jira] [Commented] (KAFKA-1567) Metric memory leaking after closing the clients

[ https://issues.apache.org/jira/browse/KAFKA-1567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14093475#comment-14093475 ] Jiangjie Qin commented on KAFKA-1567: - Updated reviewboard https://reviews.apache.org/

[jira] [Commented] (KAFKA-1485) Upgrade to Zookeeper 3.4.6

[ https://issues.apache.org/jira/browse/KAFKA-1485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14093431#comment-14093431 ] Guozhang Wang commented on KAFKA-1485: -- System tests has been a pain for long time. I

[jira] [Created] (KAFKA-1589) Strengthen System Tests

Guozhang Wang created KAFKA-1589: Summary: Strengthen System Tests Key: KAFKA-1589 URL: https://issues.apache.org/jira/browse/KAFKA-1589 Project: Kafka Issue Type: Bug Reporter: G

[jira] [Commented] (KAFKA-1485) Upgrade to Zookeeper 3.4.6

[ https://issues.apache.org/jira/browse/KAFKA-1485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14093412#comment-14093412 ] Gwen Shapira commented on KAFKA-1485: - Will take a look at the system tools. The zk j

[jira] [Comment Edited] (KAFKA-1485) Upgrade to Zookeeper 3.4.6

[ https://issues.apache.org/jira/browse/KAFKA-1485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14093381#comment-14093381 ] Joe Stein edited comment on KAFKA-1485 at 8/11/14 9:54 PM: --- I do

[jira] [Commented] (KAFKA-1485) Upgrade to Zookeeper 3.4.6

[ https://issues.apache.org/jira/browse/KAFKA-1485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14093381#comment-14093381 ] Joe Stein commented on KAFKA-1485: -- I don't think we should bind to a release not 1) stab

[jira] [Commented] (KAFKA-1485) Upgrade to Zookeeper 3.4.6

[ https://issues.apache.org/jira/browse/KAFKA-1485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14093375#comment-14093375 ] Gwen Shapira commented on KAFKA-1485: - 3.4.7 was not released yet. I'm wondering if it

[jira] [Commented] (KAFKA-1521) Producer Graceful Shutdown issue in Container (Kafka version 0.8.x.x)

[ https://issues.apache.org/jira/browse/KAFKA-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14093367#comment-14093367 ] Bravesh Mistry commented on KAFKA-1521: --- [~nehanarkhede] and [~junrao] This issue i

[jira] [Commented] (KAFKA-1567) Metric memory leaking after closing the clients

[ https://issues.apache.org/jira/browse/KAFKA-1567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14093355#comment-14093355 ] Bravesh Mistry commented on KAFKA-1567: --- HI Team, This is also related to this issu

[jira] [Commented] (KAFKA-1485) Upgrade to Zookeeper 3.4.6

[ https://issues.apache.org/jira/browse/KAFKA-1485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14093358#comment-14093358 ] Dong Lin commented on KAFKA-1485: - But is there Zookeeper release 3.4.7? From what I find

[jira] [Commented] (KAFKA-1485) Upgrade to Zookeeper 3.4.6

[ https://issues.apache.org/jira/browse/KAFKA-1485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14093347#comment-14093347 ] Gwen Shapira commented on KAFKA-1485: - Doesn't Zookeeper release 3.4.7 make more sense

[jira] [Commented] (KAFKA-1485) Upgrade to Zookeeper 3.4.6

[ https://issues.apache.org/jira/browse/KAFKA-1485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14093336#comment-14093336 ] Dong Lin commented on KAFKA-1485: - We observe that Zookeeper 3.4.6 changes its API as comp

Re: ReplicaNotAvailableCode

Thanks Joe, for fixing the missing error code. I reworded the description in the wiki a bit. I think the error code use ReplicaManager is correct. Jun On Mon, Aug 11, 2014 at 1:12 PM, Joe Stein wrote: > An issue/question just came up with a client library > https://github.com/Shopify/sarama/

Re: trunk/gradle.properties

Yes, we can do that. Thanks, Jun On Mon, Aug 11, 2014 at 1:13 PM, Joe Stein wrote: > Can we change > > version=0.8.1 > > to > > version=0.8.2-SNAPSHOT > > or something of the sort? I think it would be helpful with 0.8.2 testing. > > I want to post a build to staging maven and upload artifacts

Stable Version of New Producer on Trunk

Hi Kafka Dev, We are paining to upgrade to Java Producer (with Callback) on trunk. So questions is which tag or branch we can use for stable producer quality version. Thanks in advance for your help !! Thanks, Bhavesh

[Kafka MirrorMaker] Message with Custom Partition Logic

HI Kafka Dev Team, We have to aggregate events (count) per DC and across DCs for one of topic. We have standard Linked-in data pipe line producers --> Local Brokers --> MM --> Center Brokers. So I would like to know How MM handles messages when custom partitioning logic is used as below and

trunk/gradle.properties

Can we change version=0.8.1 to version=0.8.2-SNAPSHOT or something of the sort? I think it would be helpful with 0.8.2 testing. I want to post a build to staging maven and upload artifacts to my profile for folks so they could start to bang away at it real artifacts. I was thinking we can do

ReplicaNotAvailableCode

An issue/question just came up with a client library https://github.com/Shopify/sarama/pull/142#issuecomment-51707301 the wiki for the wire protocol had error code = 9 as unused I just updated it for ReplicaNotAvailableCode and put a description https://cwiki.apache.org/confluence/pages/diffpages.a

[jira] [Commented] (KAFKA-1460) NoReplicaOnlineException: No replica for partition

[ https://issues.apache.org/jira/browse/KAFKA-1460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14093200#comment-14093200 ] Ryan Williams commented on KAFKA-1460: -- I'm looking into this as well. Kafka was runn

Re: [New Feature Request] Ability to Inject Queue Implementation Async Mode

Thank you all for the comments. Yes, I understand concern from community members with extra burden of having the complexity to drop message, but if ability to inject implementation of the Queue which will make this completely transparent to Kafka. I just need fine-grained control of the applicati

[jira] [Commented] (KAFKA-1586) support sticky partitioning in the new producer

[ https://issues.apache.org/jira/browse/KAFKA-1586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14093185#comment-14093185 ] Jay Kreps commented on KAFKA-1586: -- We made partitioning pluggable so you could plug in t

[jira] Subscription: outstanding kafka patches

Issue Subscription Filter: outstanding kafka patches (119 issues) The list of outstanding kafka patches Subscriber: kafka-mailing-list Key Summary KAFKA-1586 support sticky partitioning in the new producer https://issues.apache.org/jira/browse/KAFKA-1586 KAFKA-1585 Client: In

[jira] [Commented] (KAFKA-1586) support sticky partitioning in the new producer

[ https://issues.apache.org/jira/browse/KAFKA-1586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14093140#comment-14093140 ] Jun Rao commented on KAFKA-1586: The question is whether sticky partitioning only helps Mi

[jira] [Commented] (KAFKA-1586) support sticky partitioning in the new producer

[ https://issues.apache.org/jira/browse/KAFKA-1586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14093110#comment-14093110 ] Guozhang Wang commented on KAFKA-1586: -- I think while we are designing the new produc

[jira] [Assigned] (KAFKA-1586) support sticky partitioning in the new producer

[ https://issues.apache.org/jira/browse/KAFKA-1586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao reassigned KAFKA-1586: -- Assignee: Jun Rao > support sticky partitioning in the new producer > -

[jira] [Updated] (KAFKA-1586) support sticky partitioning in the new producer

[ https://issues.apache.org/jira/browse/KAFKA-1586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao updated KAFKA-1586: --- Attachment: KAFKA-1586.patch > support sticky partitioning in the new producer >

[jira] [Commented] (KAFKA-1586) support sticky partitioning in the new producer

[ https://issues.apache.org/jira/browse/KAFKA-1586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14093047#comment-14093047 ] Jun Rao commented on KAFKA-1586: Created reviewboard https://reviews.apache.org/r/24565/

Review Request 24565: Patch for KAFKA-1586

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/24565/ --- Review request for kafka. Bugs: KAFKA-1586 https://issues.apache.org/jira/b

[jira] [Updated] (KAFKA-1588) Offset response does not support two requests for the same topic/partition combo

[ https://issues.apache.org/jira/browse/KAFKA-1588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Palino updated KAFKA-1588: --- Description: When performing an OffsetRequest, if you request the same topic and partition combinati

Re: Review Request 24214: Patch for KAFKA-1374

> On Aug. 10, 2014, 11:44 p.m., Jun Rao wrote: > > core/src/main/scala/kafka/log/LogCleaner.scala, lines 400-420 > > > > > > Thinking about this a bit more. I am wondering if it would be better if > > we introduce a p

[jira] [Commented] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names

[ https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14092910#comment-14092910 ] Vladimir Tretyakov commented on KAFKA-1481: --- Hi Jun Rao, look at http://search-h

[jira] [Commented] (KAFKA-1499) Broker-side compression configuration

[ https://issues.apache.org/jira/browse/KAFKA-1499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14092891#comment-14092891 ] Gwen Shapira commented on KAFKA-1499: - +1 Being able to set compression broker-side wi

[jira] [Commented] (KAFKA-1499) Broker-side compression configuration

[ https://issues.apache.org/jira/browse/KAFKA-1499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14092882#comment-14092882 ] Jun Rao commented on KAFKA-1499: Yes, I think this makes sense. The same property can be u

[jira] [Commented] (KAFKA-1588) Offset response does not support two requests for the same topic/partition combo

[ https://issues.apache.org/jira/browse/KAFKA-1588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14092870#comment-14092870 ] Jun Rao commented on KAFKA-1588: Is this the OffsetFetchRequest or OffsetRequest? > Offse

[jira] [Commented] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names

[ https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14092863#comment-14092863 ] Jun Rao commented on KAFKA-1481: I am wondering if you can get around the problem by just

[jira] [Resolved] (KAFKA-1574) unit tests can hang on socketserver shutdown

[ https://issues.apache.org/jira/browse/KAFKA-1574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao resolved KAFKA-1574. Resolution: Fixed Fix Version/s: 0.8.2 Assignee: Jun Rao Thanks for the reviews. Committed