RE: [DISCUSS] KIP-26 - Add Copycat, a connector framework for data import/export

+1 I like the idea of having CopyCat to be part of core Kafka, and having the connectors to be in a separate sub-project. This will allow CopyCat to be a new additional API for Kafka. Also it is critical to keep the CopyCat API as solid and stable as the rest of Kafka. The separate sub-projec

[jira] [Commented] (KAFKA-1367) Broker topic metadata not kept in sync with ZooKeeper

[ https://issues.apache.org/jira/browse/KAFKA-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14598880#comment-14598880 ] Ashish K Singh commented on KAFKA-1367: --- [~junrao], [~jjkoshy], [~nehanarkhede], [~g

Re: Review Request 35820: Patch for KAFKA-1367

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35820/ --- (Updated June 24, 2015, 5:10 a.m.) Review request for kafka. Bugs: KAFKA-1367

[jira] [Commented] (KAFKA-1367) Broker topic metadata not kept in sync with ZooKeeper

[ https://issues.apache.org/jira/browse/KAFKA-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14598877#comment-14598877 ] Ashish K Singh commented on KAFKA-1367: --- Created reviewboard https://reviews.apache.

[jira] [Updated] (KAFKA-1367) Broker topic metadata not kept in sync with ZooKeeper

[ https://issues.apache.org/jira/browse/KAFKA-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish K Singh updated KAFKA-1367: -- Status: Patch Available (was: Open) > Broker topic metadata not kept in sync with ZooKeeper > -

Review Request 35820: Patch for KAFKA-1367

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35820/ --- Review request for kafka. Bugs: KAFKA-1367 https://issues.apache.org/jira/b

[jira] [Updated] (KAFKA-1367) Broker topic metadata not kept in sync with ZooKeeper

[ https://issues.apache.org/jira/browse/KAFKA-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish K Singh updated KAFKA-1367: -- Attachment: KAFKA-1367.patch > Broker topic metadata not kept in sync with ZooKeeper > -

Re: [GitHub] kafka pull request: Kafka 2276

I'm unclear on the directory structure and few high level things (and I can't figure out how to comment on more than a single line in github): tests/kafkatest - isn't it redundant? do we expect any non-kafka tests? services/templates - those are basically all configuration files to be used by the

trying to manage offset for ConsumerConnector

Hi Team I am trying to create a Kafka Consumer API which is high level as ConsumerConnector but is enable to manage offset to track the message read/consumed and save from duplicate message consumption. In addition to few other pages I was going through http://www.slideshare.net/jjkoshy/offset-ma

Re: Review Request 35791: Patch for KAFKA-2298

> On June 23, 2015, 5:59 p.m., Jason Gustafson wrote: > > clients/src/main/java/org/apache/kafka/common/network/Selector.java, line > > 282 > > > > > > Do you think we should just move the disconnected.add() into the

[jira] [Commented] (KAFKA-2298) Client Selector can drop connections on InvalidReceiveException without notifying NetworkClient

[ https://issues.apache.org/jira/browse/KAFKA-2298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14598719#comment-14598719 ] Dong Lin commented on KAFKA-2298: - Updated reviewboard https://reviews.apache.org/r/35791/

[jira] [Updated] (KAFKA-2298) Client Selector can drop connections on InvalidReceiveException without notifying NetworkClient

[ https://issues.apache.org/jira/browse/KAFKA-2298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Lin updated KAFKA-2298: Attachment: KAFKA-2298_2015-06-23_18:47:54.patch > Client Selector can drop connections on InvalidReceiveExc

Re: Review Request 35791: Patch for KAFKA-2298

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35791/ --- (Updated June 24, 2015, 1:48 a.m.) Review request for kafka. Bugs: KAFKA-2298

Re: Review Request 35791: Patch for KAFKA-2298

> On June 23, 2015, 5:59 p.m., Jason Gustafson wrote: > > clients/src/main/java/org/apache/kafka/common/network/Selector.java, line > > 282 > > > > > > Do you think we should just move the disconnected.add() into the

Re: [DISCUSS] KIP-26 - Add Copycat, a connector framework for data import/export

One more reason to have CopyCat as a separate project is to sidestep the entire "Why CopyCat and not X" discussion :) On Tue, Jun 23, 2015 at 6:26 PM, Gwen Shapira wrote: > Re: Flume vs. CopyCat > > I would love to have an automagically-parallelizing, schema-aware > version of Flume with great re

Re: [DISCUSS] KIP-26 - Add Copycat, a connector framework for data import/export

Re: Flume vs. CopyCat I would love to have an automagically-parallelizing, schema-aware version of Flume with great reliability guarantees. Flume has good core architecture and I'm sure that if the Flume community is interested, it can be extended in that direction. However, the Apache way is not

Re: [DISCUSS] KIP-26 - Add Copycat, a connector framework for data import/export

I am still not convinced why a stream processing framework closely tied to Kafka will not help with this (since we are also referring to some basic transformations). The devil is in the details of the design and I would be able to better comment on it after that. I would love to see a detailed desi

Re: [DISCUSS] KIP-26 - Add Copycat, a connector framework for data import/export

And, one more piece of follow up. Some folks were wondering about more specific details about what we had in mind for the framework. Along with a prototype I had been writing up some documentation. This isn't meant in any way to be finalized and I just wrote it up using the same tools we use intern

Re: [DISCUSS] KIP-26 - Add Copycat, a connector framework for data import/export

On Mon, Jun 22, 2015 at 8:32 PM, Roshan Naik wrote: > Thanks Jay and Ewen for the response. > > > >@Jay > > > > 3. This has a built in notion of parallelism throughout. > > > > It was not obvious how it will look like or differ from existing systemsŠ > since all of existing ones do parallelize da

[jira] [Commented] (KAFKA-1215) Rack-Aware replica assignment option

[ https://issues.apache.org/jira/browse/KAFKA-1215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14598643#comment-14598643 ] Jay Kreps commented on KAFKA-1215: -- This is great! > Rack-Aware replica assignment optio

[jira] [Commented] (KAFKA-1215) Rack-Aware replica assignment option

[ https://issues.apache.org/jira/browse/KAFKA-1215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14598615#comment-14598615 ] Allen Wang commented on KAFKA-1215: --- We have a working solution now for rack aware assig

Re: [DISCUSS] KIP-26 - Add Copycat, a connector framework for data import/export

There was some discussion on the KIP call today. I'll give my summary of what I heard here to make sure this thread has the complete context for ongoing discussion. * Where the project should live, and if in Kafka, where should connectors live? If some are in Kafka and some not, how many and which

[jira] [Updated] (KAFKA-2298) Client Selector can drop connections on InvalidReceiveException without notifying NetworkClient

[ https://issues.apache.org/jira/browse/KAFKA-2298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Lin updated KAFKA-2298: Labels: quotas (was: ) > Client Selector can drop connections on InvalidReceiveException without > notifyi

Re: Review Request 33620: Patch for KAFKA-1690

> On May 22, 2015, 12:33 a.m., Jun Rao wrote: > > clients/src/main/java/org/apache/kafka/common/network/SSLTransportLayer.java, > > line 294 > > > > > > Not sure why we are looping back here. If the HandshakeStatus is

Re: Review Request 33620: Patch for KAFKA-1690

> On May 22, 2015, 12:33 a.m., Jun Rao wrote: > > clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java, > > line 231 > > > > > > Do we need config.values? Could we just pass in config.originals()

[jira] [Updated] (KAFKA-1690) new java producer needs ssl support as a client

[ https://issues.apache.org/jira/browse/KAFKA-1690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sriharsha Chintalapani updated KAFKA-1690: -- Attachment: KAFKA-1690_2015-06-23_13:18:20.patch > new java producer needs ssl s

Re: Review Request 33620: Patch for KAFKA-1690

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33620/ --- (Updated June 23, 2015, 8:18 p.m.) Review request for kafka. Bugs: KAFKA-1690

[jira] [Commented] (KAFKA-1690) new java producer needs ssl support as a client

[ https://issues.apache.org/jira/browse/KAFKA-1690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14598297#comment-14598297 ] Sriharsha Chintalapani commented on KAFKA-1690: --- Updated reviewboard https:/

Re: Review Request 35791: Patch for KAFKA-2298

> On June 23, 2015, 5:59 p.m., Jason Gustafson wrote: > > clients/src/main/java/org/apache/kafka/common/network/Selector.java, line > > 282 > > > > > > Do you think we should just move the disconnected.add() into the

Re: Review Request 35791: Patch for KAFKA-2298

> On June 23, 2015, 5:59 p.m., Jason Gustafson wrote: > > clients/src/main/java/org/apache/kafka/common/network/Selector.java, line > > 282 > > > > > > Do you think we should just move the disconnected.add() into the

RE: [VOTE] KIP-23 - Add JSON/CSV output and looping options to ConsumerGroupCommand

Thanks. +1 (non-binding) From: Ashish Singh [asi...@cloudera.com] Sent: Tuesday, June 23, 2015 12:04 PM To: dev@kafka.apache.org Subject: Re: [VOTE] KIP-23 - Add JSON/CSV output and looping options to ConsumerGroupCommand Aditya, Yes. Updated the KIP wit

Re: [VOTE] KIP-23 - Add JSON/CSV output and looping options to ConsumerGroupCommand

Aditya, Yes. Updated the KIP with notes on the parameter. Will update the patch. On Tue, Jun 23, 2015 at 11:51 AM, Aditya Auradkar < aaurad...@linkedin.com.invalid> wrote: > Hey Ashish, > > Last hangout, I think we discussed adding a "num-iterations" parameter to > the script. Do you plan to sup

Re: Review Request 35791: Patch for KAFKA-2298

> On June 23, 2015, 5:59 p.m., Jason Gustafson wrote: > > clients/src/main/java/org/apache/kafka/common/network/Selector.java, line > > 282 > > > > > > Do you think we should just move the disconnected.add() into the

Re: [VOTE] KIP-23 - Add JSON/CSV output and looping options to ConsumerGroupCommand

+1 (non-binding) On Tue, Jun 23, 2015 at 11:24 AM, Neha Narkhede wrote: > +1 > > On Tue, Jun 23, 2015 at 11:15 AM, Ashish Singh wrote: > >> Hey Guys, >> >> We had some discussion over mail and in KIP hangout. I will update the RB >> with proposed changes. >> >> >> On Sun, Jun 14, 2015 at 10:07 A

RE: [VOTE] KIP-23 - Add JSON/CSV output and looping options to ConsumerGroupCommand

Hey Ashish, Last hangout, I think we discussed adding a "num-iterations" parameter to the script. Do you plan to support that? Aditya From: Neha Narkhede [n...@confluent.io] Sent: Tuesday, June 23, 2015 11:24 AM To: dev@kafka.apache.org Cc: Jun Rao Subje

Re: Review Request 35791: Patch for KAFKA-2298

> On June 23, 2015, 5:59 p.m., Jason Gustafson wrote: > > clients/src/main/java/org/apache/kafka/common/network/Selector.java, line > > 282 > > > > > > Do you think we should just move the disconnected.add() into the

Re: [VOTE] KIP-23 - Add JSON/CSV output and looping options to ConsumerGroupCommand

+1 On Tue, Jun 23, 2015 at 11:15 AM, Ashish Singh wrote: > Hey Guys, > > We had some discussion over mail and in KIP hangout. I will update the RB > with proposed changes. > > > On Sun, Jun 14, 2015 at 10:07 AM, Ashish Singh > wrote: > > > Hi Neha, > > > > Answers inline. > > > > On Thu, Jun 11

high level consumer memory footprint

Hi, I was just wondering if there is any difference in the memory footprint of a high level consumer when: 1. the consumer is live and continuously consuming messages with no backlogs 2. when the consumer is down for quite some time and needs to be brought up to clear the backlog. My test case w

[VOTE] KIP-23 - Add JSON/CSV output and looping options to ConsumerGroupCommand

Hey Guys, We had some discussion over mail and in KIP hangout. I will update the RB with proposed changes. On Sun, Jun 14, 2015 at 10:07 AM, Ashish Singh wrote: > Hi Neha, > > Answers inline. > > On Thu, Jun 11, 2015 at 7:20 PM, Neha Narkhede wrote: > >> Thanks for submitting the KIP, Ashish!

Re: Review Request 35791: Patch for KAFKA-2298

> On June 23, 2015, 5:59 p.m., Jason Gustafson wrote: > > clients/src/main/java/org/apache/kafka/common/network/Selector.java, line > > 282 > > > > > > Do you think we should just move the disconnected.add() into the

Re: Review Request 35791: Patch for KAFKA-2298

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35791/#review89010 --- clients/src/main/java/org/apache/kafka/common/network/Selector.java

[jira] [Resolved] (KAFKA-2294) javadoc compile error due to illegal , build failing (jdk 8)

[ https://issues.apache.org/jira/browse/KAFKA-2294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jakob Homan resolved KAFKA-2294. Resolution: Fixed Fix Version/s: 0.9.0 Reviewer: Jakob Homan Assignee: Jeff

[jira] [Commented] (KAFKA-2294) javadoc compile error due to illegal , build failing (jdk 8)

[ https://issues.apache.org/jira/browse/KAFKA-2294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14598057#comment-14598057 ] Jakob Homan commented on KAFKA-2294: +1. > javadoc compile error due to illegal , bu

Re: [GitHub] kafka pull request: Kafka 2276

Awesome, thanks :) On Tue, Jun 23, 2015 at 10:32 AM, Geoffrey Anderson wrote: > Hi Gwen, > > That is indeed the plan, and my understanding is that the merge script > Ismael has been working on helps committers with this step. > > I'm trying out the Github flow roughly as outlined here: > http://m

[jira] [Updated] (KAFKA-2298) Client Selector can drop connections on InvalidReceiveException without notifying NetworkClient

[ https://issues.apache.org/jira/browse/KAFKA-2298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Lin updated KAFKA-2298: Labels: (was: quotas) > Client Selector can drop connections on InvalidReceiveException without > notifyi

[jira] [Updated] (KAFKA-2298) Client Selector can drop connections on InvalidReceiveException without notifying NetworkClient

[ https://issues.apache.org/jira/browse/KAFKA-2298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Lin updated KAFKA-2298: Attachment: KAFKA-2298.patch > Client Selector can drop connections on InvalidReceiveException without > no

[jira] [Updated] (KAFKA-2298) Client Selector can drop connections on InvalidReceiveException without notifying NetworkClient

[ https://issues.apache.org/jira/browse/KAFKA-2298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Lin updated KAFKA-2298: Status: Patch Available (was: Open) > Client Selector can drop connections on InvalidReceiveException witho

[jira] [Commented] (KAFKA-2298) Client Selector can drop connections on InvalidReceiveException without notifying NetworkClient

[ https://issues.apache.org/jira/browse/KAFKA-2298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14598024#comment-14598024 ] Dong Lin commented on KAFKA-2298: - Created reviewboard https://reviews.apache.org/r/35791/

Review Request 35791: Patch for KAFKA-2298

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35791/ --- Review request for kafka. Bugs: KAFKA-2298 https://issues.apache.org/jira/b

[jira] [Created] (KAFKA-2298) Client Selector can drop connections on InvalidReceiveException without notifying NetworkClient

Dong Lin created KAFKA-2298: --- Summary: Client Selector can drop connections on InvalidReceiveException without notifying NetworkClient Key: KAFKA-2298 URL: https://issues.apache.org/jira/browse/KAFKA-2298 P

Re: [GitHub] kafka pull request: Kafka 2276

Hi Gwen, That is indeed the plan, and my understanding is that the merge script Ismael has been working on helps committers with this step. I'm trying out the Github flow roughly as outlined here: http://mail-archives.apache.org/mod_mbox/kafka-dev/201504.mbox/%3ccad5tkzab-hkey-zcr8x4wtxawybxpojx6

[jira] [Commented] (KAFKA-2297) VerifiableProperties fails when constructed with Properties with numeric values

[ https://issues.apache.org/jira/browse/KAFKA-2297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14597942#comment-14597942 ] Gwen Shapira commented on KAFKA-2297: - Yeah, I think its a known issue - VerifiablePro

[jira] [Created] (KAFKA-2297) VerifiableProperties fails when constructed with Properties with numeric values

Grant Overby created KAFKA-2297: --- Summary: VerifiableProperties fails when constructed with Properties with numeric values Key: KAFKA-2297 URL: https://issues.apache.org/jira/browse/KAFKA-2297 Project:

[jira] [Updated] (KAFKA-2168) New consumer poll() can block other calls like position(), commit(), and close() indefinitely

[ https://issues.apache.org/jira/browse/KAFKA-2168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Gustafson updated KAFKA-2168: --- Attachment: KAFKA-2168_2015-06-23_09:39:07.patch > New consumer poll() can block other calls l

Re: Review Request 34789: Patch for KAFKA-2168

> On June 23, 2015, 4:20 a.m., Jun Rao wrote: > > clients/src/main/java/org/apache/kafka/clients/consumer/internals/Coordinator.java, > > line 355 > > > > > > What's the logic to initiate connection to coordinator if

[jira] [Commented] (KAFKA-2168) New consumer poll() can block other calls like position(), commit(), and close() indefinitely

[ https://issues.apache.org/jira/browse/KAFKA-2168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14597914#comment-14597914 ] Jason Gustafson commented on KAFKA-2168: Updated reviewboard https://reviews.apach

Re: Review Request 34789: Patch for KAFKA-2168

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/34789/ --- (Updated June 23, 2015, 4:39 p.m.) Review request for kafka. Bugs: KAFKA-2168

[jira] [Updated] (KAFKA-2294) javadoc compile error due to illegal , build failing (jdk 8)

[ https://issues.apache.org/jira/browse/KAFKA-2294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Maxwell updated KAFKA-2294: Attachment: KAFKA-2294-1.patch Fixed close() javadoc > javadoc compile error due to illegal , buil

Re: Review Request 33620: Patch for KAFKA-1690

> On May 22, 2015, 12:33 a.m., Jun Rao wrote: > > clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java, > > line 231 > > > > > > Do we need config.values? Could we just pass in config.originals()

[jira] [Comment Edited] (KAFKA-1901) Move Kafka version to be generated in code by build (instead of in manifest)

[ https://issues.apache.org/jira/browse/KAFKA-1901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14597558#comment-14597558 ] Manikumar Reddy edited comment on KAFKA-1901 at 6/23/15 12:13 PM: --

[jira] [Commented] (KAFKA-1901) Move Kafka version to be generated in code by build (instead of in manifest)

[ https://issues.apache.org/jira/browse/KAFKA-1901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14597558#comment-14597558 ] Manikumar Reddy commented on KAFKA-1901: Probably the easiest thing to do would be

[jira] [Comment Edited] (KAFKA-1901) Move Kafka version to be generated in code by build (instead of in manifest)

[ https://issues.apache.org/jira/browse/KAFKA-1901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14597558#comment-14597558 ] Manikumar Reddy edited comment on KAFKA-1901 at 6/23/15 12:10 PM: --

[jira] [Created] (KAFKA-2296) Not able to delete topic on latest kafka

Andrew M created KAFKA-2296: --- Summary: Not able to delete topic on latest kafka Key: KAFKA-2296 URL: https://issues.apache.org/jira/browse/KAFKA-2296 Project: Kafka Issue Type: Bug Affects Vers

[jira] [Created] (KAFKA-2295) Dynamically loaded classes (encoders, etc.) may not be found by Kafka Producer

Tathagata Das created KAFKA-2295: Summary: Dynamically loaded classes (encoders, etc.) may not be found by Kafka Producer Key: KAFKA-2295 URL: https://issues.apache.org/jira/browse/KAFKA-2295 Project