Re: [ANNOUNCE] New committer: Boyang Chen

2020-06-24 Thread Liquan Pei
ode contributions, Boyang > > has > > > also helped reviewing even more PRs and KIPs than his own. > > > > > > Thanks for all the contributions Boyang! And look forward to more > > > collaborations with you on Apache Kafka. > > > > > > > > > -- Guozhang, on behalf of the Apache Kafka PMC > > > > > > -- Liquan Pei Software Engineer, Confluent Inc

Re: [VOTE] KIP-429: Kafka Consumer Incremental Rebalance Protocol

2019-05-22 Thread Liquan Pei
col#KIP-429:KafkaConsumerIncrementalRebalanceProtocol-RebalanceCallbackErrorHandling > > And the on-going PRs available for review: > > Part I: https://github.com/apache/kafka/pull/6528 > Part II: https://github.com/apache/kafka/pull/6778 > > > Thanks > -- Guozhang > -- Liquan Pei Software Engineer, Confluent Inc

Re: Vote for KIP-393 (Fix time windowed serde to deserialize changelog topic)

2018-12-06 Thread Liquan Pei
> > I wanted to start a vote on approval of KIP-393 > > < > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-393%3A+Time+windowed+serde+to+properly+deserialize+changelog+input+topic > > > > > to > > fix the current time windowed serde for properly deserializing changelog > > input topics. Let me know what you guys think. > > > > Thanks, > > Shawn > > > > > -- > -- Guozhang > -- Liquan Pei Software Engineer, Confluent Inc

[jira] [Created] (KAFKA-7105) Refactor RocksDBSegmentsBatchingRestoreCallback and RocksDBBatchingRestoreCallback into a single class

2018-06-26 Thread Liquan Pei (JIRA)
Liquan Pei created KAFKA-7105: - Summary: Refactor RocksDBSegmentsBatchingRestoreCallback and RocksDBBatchingRestoreCallback into a single class Key: KAFKA-7105 URL: https://issues.apache.org/jira/browse/KAFKA-7105

[jira] [Created] (KAFKA-7103) Use bulkloading for RocksDBSegmentedBytesStore during init

2018-06-26 Thread Liquan Pei (JIRA)
Liquan Pei created KAFKA-7103: - Summary: Use bulkloading for RocksDBSegmentedBytesStore during init Key: KAFKA-7103 URL: https://issues.apache.org/jira/browse/KAFKA-7103 Project: Kafka Issue

[jira] [Created] (KAFKA-7023) Kafka Streams RocksDB bulk loading config may not be honored with customized RocksDBConfigSetter

2018-06-07 Thread Liquan Pei (JIRA)
Liquan Pei created KAFKA-7023: - Summary: Kafka Streams RocksDB bulk loading config may not be honored with customized RocksDBConfigSetter Key: KAFKA-7023 URL: https://issues.apache.org/jira/browse/KAFKA-7023

Re: [DISCUSSION] KIP-300: Add Windowed KTable API in StreamsBuilder

2018-05-21 Thread Liquan Pei
derstood that the community is busy working on 2.0 release, but this > KIP is really important for our internal use case. So if any of you got > time, please focus on clarifying the use case and reaching the agreement of > API. Really appreciate your time! > > > Best, > > Boyang > > > -- Liquan Pei Software Engineer, Confluent Inc

[jira] [Assigned] (KAFKA-4136) KafkaBasedLog should include offsets it is trying to/successfully read to in log messages

2016-09-08 Thread Liquan Pei (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liquan Pei reassigned KAFKA-4136: - Assignee: Liquan Pei (was: Ewen Cheslack-Postava) > KafkaBasedLog should include offsets it

[jira] [Created] (KAFKA-4002) task.open() should be invoked in case that 0 partitions is assigned to task.

2016-07-27 Thread Liquan Pei (JIRA)
Liquan Pei created KAFKA-4002: - Summary: task.open() should be invoked in case that 0 partitions is assigned to task. Key: KAFKA-4002 URL: https://issues.apache.org/jira/browse/KAFKA-4002 Project: Kafka

[jira] [Created] (KAFKA-3920) Add Schema source connector to Kafka Connect

2016-06-29 Thread Liquan Pei (JIRA)
Liquan Pei created KAFKA-3920: - Summary: Add Schema source connector to Kafka Connect Key: KAFKA-3920 URL: https://issues.apache.org/jira/browse/KAFKA-3920 Project: Kafka Issue Type: Improvement

[jira] [Assigned] (KAFKA-3868) New producer metric record-size-avg does not provide average record size as advertised

2016-06-20 Thread Liquan Pei (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liquan Pei reassigned KAFKA-3868: - Assignee: Liquan Pei > New producer metric record-size-avg does not provide average record s

[jira] [Assigned] (KAFKA-3829) Warn that kafka-connect group.id must not conflict with connector names

2016-06-20 Thread Liquan Pei (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liquan Pei reassigned KAFKA-3829: - Assignee: Liquan Pei (was: Ewen Cheslack-Postava) > Warn that kafka-connect group.id must

[jira] [Assigned] (KAFKA-3820) Provide utilities for tracking source offsets

2016-06-09 Thread Liquan Pei (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liquan Pei reassigned KAFKA-3820: - Assignee: Liquan Pei (was: Ewen Cheslack-Postava) > Provide utilities for tracking sou

[jira] [Updated] (KAFKA-3782) Transient failure with kafkatest.tests.connect.connect_distributed_test.ConnectDistributedTest.test_bounce.clean=True

2016-06-02 Thread Liquan Pei (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liquan Pei updated KAFKA-3782: -- Description: For commit 946ae60 max() arg is an empty sequence Traceback (most recent call last

[jira] [Created] (KAFKA-3782) Transient failure with kafkatest.tests.connect.connect_distributed_test.ConnectDistributedTest.test_bounce.clean=True

2016-06-02 Thread Liquan Pei (JIRA)
Liquan Pei created KAFKA-3782: - Summary: Transient failure with kafkatest.tests.connect.connect_distributed_test.ConnectDistributedTest.test_bounce.clean=True Key: KAFKA-3782 URL: https://issues.apache.org/jira

[jira] [Assigned] (KAFKA-2894) WorkerSinkTask doesn't handle rewinding offsets on rebalance

2016-05-31 Thread Liquan Pei (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liquan Pei reassigned KAFKA-2894: - Assignee: Liquan Pei (was: Ewen Cheslack-Postava) > WorkerSinkTask doesn't handle r

Re: kafka consumer group is rebalancing

2016-05-31 Thread Liquan Pei
s > as 3 and heartbeat.interval.ms as 1 to consumer and polling > happens > for sure with in 3. Can anyone help me out. Please let me know if any > information is needed. > I am using 3 node kafka cluster. > Thanks, > Sunny > -- Liquan Pei Software Engineer, Confluent Inc

Re: Kafka Connect connector monitoring tasks via offset storage reader

2016-05-24 Thread Liquan Pei
> could use the OffsetStorageReader to read previously-recorded offsets to > more intelligently configure its tasks. This seems very straightforward, > backward compatible, and non-intrusive. > > Is there any interest in this? If so, I can create an issue and work on a > pull req

Re: [DISCUSS] KIP-48 Support for delegation tokens as an authentication mechanism

2016-05-24 Thread Liquan Pei
It seems that the links to images in the KIP are broken. Liquan On Tue, May 24, 2016 at 9:33 AM, parth brahmbhatt < brahmbhatt.pa...@gmail.com> wrote: > 110. What does getDelegationTokenAs mean? > In the current proposal we only allow a user to get delegation token for > the identity that it aut

Re: [DISCUSS] scalability limits in the coordinator

2016-05-23 Thread Liquan Pei
>>>>> 60 consumers 1598621 bytes > > >>>>>>> 70 consumers 1837359 bytes > > >>>>>>> 80 consumers 2066934 bytes > > >>>>>>> 90 consumers 2310970 bytes > > >>>>>>> 100 consumers 2542735 bytes > > >>>>>>> > > >>>>>>> Note that the growth itself is pretty gradual. Plotting the > > >> points > > >>>>> makes > > >>>>>> it > > >>>>>>> look roughly linear w.r.t the number of consumers: > > >>>>>>> > > >>>>>>> > > >>>>>> > > >>>>> > > >>>> > > >>> > > >> > > > https://www.wolframalpha.com/input/?i=(1,+54739),+(5,+261524),+(10,+459804),+(20,+702499),+(30,+930525),+(40,+1115657),+(50,+1363112),+(60,+1598621),+(70,+1837359),+(80,+2066934),+(90,+2310970),+(100,+2542735) > > >>>>>>> > > >>>>>>> Also note that these numbers aren't averages or medians or > > >> anything > > >>>>> like > > >>>>>>> that. It's just the byte size from a given run. I did run them a > > >>> few > > >>>>>> times > > >>>>>>> and saw similar results. > > >>>>>>> > > >>>>>>> Impact: > > >>>>>>> Even after adding gzip to the __consumer_offsets topic with my > > >>>> pending > > >>>>>>> KAFKA-3718 patch, the AwaitingSync phase of the group fails with > > >>>>>>> RecordTooLargeException. This means the combined size of each > > >>>> member's > > >>>>>>> subscriptions and assignments exceeded the > > >>>> KafkaConfig.messageMaxBytes > > >>>>> of > > >>>>>>> 112 bytes. The group ends up dying. > > >>>>>>> > > >>>>>>> Options: > > >>>>>>> 1. Config change: reduce the number of consumers in the group. > > >> This > > >>>>> isn't > > >>>>>>> always a realistic answer in more strenuous use cases like > > >>>> MirrorMaker > > >>>>>>> clusters or for auditing. > > >>>>>>> 2. Config change: split the group into smaller groups which > > >>> together > > >>>>> will > > >>>>>>> get full coverage of the topics. This gives each group member a > > >>>> smaller > > >>>>>>> subscription.(ex: g1 has topics starting with a-m while g2 has > > >>> topics > > >>>>>>> starting ith n-z). This would be operationally painful to manage. > > >>>>>>> 3. Config change: split the topics among members of the group. > > >>> Again > > >>>>> this > > >>>>>>> gives each group member a smaller subscription. This would also > > >> be > > >>>>>>> operationally painful to manage. > > >>>>>>> 4. Config change: bump up KafkaConfig.messageMaxBytes (a > > >>> topic-level > > >>>>>>> config) and KafkaConfig.replicaFetchMaxBytes (a broker-level > > >>> config). > > >>>>>>> Applying messageMaxBytes to just the __consumer_offsets topic > > >> seems > > >>>>>>> relatively harmless, but bumping up the broker-level > > >>>>> replicaFetchMaxBytes > > >>>>>>> would probably need more attention. > > >>>>>>> 5. Config change: try different compression codecs. Based on 2 > > >>>> minutes > > >>>>> of > > >>>>>>> googling, it seems like lz4 and snappy are faster than gzip but > > >>> have > > >>>>>> worse > > >>>>>>> compression, so this probably won't help. > > >>>>>>> 6. Implementation change: support sending the regex over the wire > > >>>>> instead > > >>>>>>> of the fully expanded topic subscriptions. I think people said in > > >>> the > > >>>>>> past > > >>>>>>> that different languages have subtle differences in regex, so > > >> this > > >>>>>> doesn't > > >>>>>>> play nicely with cross-language groups. > > >>>>>>> 7. Implementation change: maybe we can reverse the mapping? > > >> Instead > > >>>> of > > >>>>>>> mapping from member to subscriptions, we can map a subscription > > >> to > > >>> a > > >>>>> list > > >>>>>>> of members. > > >>>>>>> 8. Implementation change: maybe we can try to break apart the > > >>>>>> subscription > > >>>>>>> and assignments from the same SyncGroupRequest into multiple > > >>> records? > > >>>>>> They > > >>>>>>> can still go to the same message set and get appended together. > > >>> This > > >>>>> way > > >>>>>>> the limit become the segment size, which shouldn't be a problem. > > >>> This > > >>>>> can > > >>>>>>> be tricky to get right because we're currently keying these > > >>> messages > > >>>> on > > >>>>>> the > > >>>>>>> group, so I think records from the same rebalance might > > >>> accidentally > > >>>>>>> compact one another, but my understanding of compaction isn't > > >> that > > >>>>> great. > > >>>>>>> > > >>>>>>> Todo: > > >>>>>>> It would be interesting to rerun the tests with no compression > > >> just > > >>>> to > > >>>>>> see > > >>>>>>> how much gzip is helping but it's getting late. Maybe tomorrow? > > >>>>>>> > > >>>>>>> - Onur > > >>>>>>> > > >>>>>> > > >>>>> > > >>>> > > >>>> > > >>>> > > >>>> -- > > >>>> -- Guozhang > > >>>> > > >>> > > >> > > >> > > >> > > >> -- > > >> -- Guozhang > > >> > > > > > -- Liquan Pei Software Engineer, Confluent Inc

[jira] [Updated] (KAFKA-3742) Can't run connect-distributed.sh with -daemon flag

2016-05-20 Thread Liquan Pei (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liquan Pei updated KAFKA-3742: -- Description: Running on ubuntu 14.04. Discovered while experimenting various different kafka

[jira] [Updated] (KAFKA-3742) Can't run connect-distributed.sh with -daemon flag

2016-05-20 Thread Liquan Pei (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liquan Pei updated KAFKA-3742: -- Summary: Can't run connect-distributed.sh with -daemon flag (was: Can't run connect-distri

[jira] [Assigned] (KAFKA-3742) Can't run connect-distributed with -daemon flag

2016-05-20 Thread Liquan Pei (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liquan Pei reassigned KAFKA-3742: - Assignee: Liquan Pei > Can't run connect-distributed with -dae

[jira] [Assigned] (KAFKA-3739) Add no-arg constructor for library provided serdes

2016-05-20 Thread Liquan Pei (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liquan Pei reassigned KAFKA-3739: - Assignee: Liquan Pei > Add no-arg constructor for library provided ser

[jira] [Assigned] (KAFKA-3734) Exceptions in SourceTask.commit() methods should cause task to stop and report failed state

2016-05-19 Thread Liquan Pei (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liquan Pei reassigned KAFKA-3734: - Assignee: Liquan Pei (was: Ewen Cheslack-Postava) > Exceptions in SourceTask.commit() meth

[jira] [Assigned] (KAFKA-3705) Support non-key joining in KTable

2016-05-12 Thread Liquan Pei (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liquan Pei reassigned KAFKA-3705: - Assignee: Liquan Pei (was: Guozhang Wang) > Support non-key joining in KTa

[jira] [Updated] (KAFKA-3692) Wildcards in External CLASSPATH may cause it not be included in the CLASSPATH

2016-05-11 Thread Liquan Pei (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liquan Pei updated KAFKA-3692: -- Fix Version/s: (was: 0.10.0.1) 0.10.0.0 > Wildcards in External CLASSPATH

[jira] [Updated] (KAFKA-3692) Wildcards in External CLASSPATH may cause it not be included in the CLASSPATH

2016-05-11 Thread Liquan Pei (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liquan Pei updated KAFKA-3692: -- Priority: Blocker (was: Major) > Wildcards in External CLASSPATH may cause it not be included in

Re: [VOTE] 0.10.0.0 RC4

2016-05-10 Thread Liquan Pei
tml > > * Protocol: > http://kafka.apache.org/0100/protocol.html > > /** > > Thanks, > > Gwen > -- Liquan Pei Software Engineer, Confluent Inc

[jira] [Updated] (KAFKA-3692) Wildcards in External CLASSPATH may cause it not be included in the CLASSPATH

2016-05-10 Thread Liquan Pei (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liquan Pei updated KAFKA-3692: -- Description: Currently, we doesn't use double quote when using CLASSPATH in kafka-run-class.sh.

[jira] [Updated] (KAFKA-3692) Wildcards in External CLASSPATH may cause it not be included in the CLASSPATH

2016-05-10 Thread Liquan Pei (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liquan Pei updated KAFKA-3692: -- Summary: Wildcards in External CLASSPATH may cause it not be included in the CLASSPATH (was: Spaces in

[jira] [Commented] (KAFKA-3690) Avoid passing null to UnmodifiableMap

2016-05-10 Thread Liquan Pei (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15279352#comment-15279352 ] Liquan Pei commented on KAFKA-3690: --- If JsonConverter is used for {

[jira] [Updated] (KAFKA-3692) Spaces in External CLASSPATH may cause it not be included in the CLASSPATH

2016-05-10 Thread Liquan Pei (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liquan Pei updated KAFKA-3692: -- Status: Patch Available (was: In Progress) > Spaces in External CLASSPATH may cause it not be inclu

[jira] [Updated] (KAFKA-3692) Spaces in External CLASSPATH may cause it not be included in the CLASSPATH

2016-05-10 Thread Liquan Pei (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liquan Pei updated KAFKA-3692: -- Fix Version/s: (was: 0.10.0.0) 0.10.0.1 > Spaces in External CLASSPATH may ca

[jira] [Updated] (KAFKA-3692) Spaces in External CLASSPATH may cause it not be included in the CLASSPATH

2016-05-10 Thread Liquan Pei (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liquan Pei updated KAFKA-3692: -- Description: Currently, we doesn't use double quote when using CLASSPATH in kafka-run-class.sh.

[jira] [Updated] (KAFKA-3692) Spaces in External CLASSPATH may cause it not be included in the CLASSPATH

2016-05-10 Thread Liquan Pei (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liquan Pei updated KAFKA-3692: -- Description: Currently, we doesn't use double quote when using CLASSPATH in (was: This may

[jira] [Work started] (KAFKA-3692) Spaces in External CLASSPATH may cause it not be included in the CLASSPATH

2016-05-10 Thread Liquan Pei (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on KAFKA-3692 started by Liquan Pei. - > Spaces in External CLASSPATH may cause it not be included in the CLASSP

[jira] [Commented] (KAFKA-3690) Avoid passing null to UnmodifiableMap

2016-05-10 Thread Liquan Pei (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15279281#comment-15279281 ] Liquan Pei commented on KAFKA-3690: --- Agreed with [~ewencp]. I think we can

[jira] [Updated] (KAFKA-3690) Avoid passing null to UnmodifiableMap

2016-05-10 Thread Liquan Pei (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liquan Pei updated KAFKA-3690: -- Summary: Avoid passing null to UnmodifiableMap (was: Avoid passing null to UnModifiableMap) > Av

[jira] [Updated] (KAFKA-3690) Avoid passing null to UnmodifiableMap

2016-05-10 Thread Liquan Pei (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liquan Pei updated KAFKA-3690: -- Priority: Major (was: Blocker) > Avoid passing null to Unmodifiable

[jira] [Updated] (KAFKA-3692) Spaces in External CLASSPATH may cause it not be included in the CLASSPATH

2016-05-10 Thread Liquan Pei (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liquan Pei updated KAFKA-3692: -- Priority: Major (was: Blocker) > Spaces in External CLASSPATH may cause it not be included in

[jira] [Created] (KAFKA-3692) Spaces in External CLASSPATH may cause it not be included in the CLASSPATH

2016-05-10 Thread Liquan Pei (JIRA)
Liquan Pei created KAFKA-3692: - Summary: Spaces in External CLASSPATH may cause it not be included in the CLASSPATH Key: KAFKA-3692 URL: https://issues.apache.org/jira/browse/KAFKA-3692 Project: Kafka

[jira] [Work started] (KAFKA-3690) Avoid passing null to UnModifiableMap

2016-05-10 Thread Liquan Pei (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on KAFKA-3690 started by Liquan Pei. - > Avoid passing null to UnModifiable

[jira] [Created] (KAFKA-3690) Avoid passing null pointers to UnModifiableMap

2016-05-10 Thread Liquan Pei (JIRA)
Liquan Pei created KAFKA-3690: - Summary: Avoid passing null pointers to UnModifiableMap Key: KAFKA-3690 URL: https://issues.apache.org/jira/browse/KAFKA-3690 Project: Kafka Issue Type: Bug

[jira] [Updated] (KAFKA-3690) Avoid passing null to UnModifiableMap

2016-05-10 Thread Liquan Pei (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liquan Pei updated KAFKA-3690: -- Summary: Avoid passing null to UnModifiableMap (was: Avoid passing null pointers to UnModifiableMap

[jira] [Work started] (KAFKA-3684) Not all SinkConnectorConfig configs are returned

2016-05-09 Thread Liquan Pei (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on KAFKA-3684 started by Liquan Pei. - > Not all SinkConnectorConfig configs are retur

[jira] [Updated] (KAFKA-3684) Not all SinkConnectorConfig configs are returned

2016-05-09 Thread Liquan Pei (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liquan Pei updated KAFKA-3684: -- Status: Patch Available (was: In Progress) > Not all SinkConnectorConfig configs are retur

[jira] [Created] (KAFKA-3684) Not all SinkConnectorConfig configs are returned

2016-05-09 Thread Liquan Pei (JIRA)
Liquan Pei created KAFKA-3684: - Summary: Not all SinkConnectorConfig configs are returned Key: KAFKA-3684 URL: https://issues.apache.org/jira/browse/KAFKA-3684 Project: Kafka Issue Type: Bug

[jira] [Updated] (KAFKA-3673) Connect tests dont handle concurrent config changes

2016-05-06 Thread Liquan Pei (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liquan Pei updated KAFKA-3673: -- Description: [INFO - 2016-04-28 18:42:45,346 - runner - log - lineno:221]: SerialTestRunner

[jira] [Created] (KAFKA-3673) Connect tests dont handle concurrent config changes

2016-05-06 Thread Liquan Pei (JIRA)
Liquan Pei created KAFKA-3673: - Summary: Connect tests dont handle concurrent config changes Key: KAFKA-3673 URL: https://issues.apache.org/jira/browse/KAFKA-3673 Project: Kafka Issue Type: Bug

[jira] [Work started] (KAFKA-3671) Topics should not be in common ConnectorConfig definitions

2016-05-06 Thread Liquan Pei (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on KAFKA-3671 started by Liquan Pei. - > Topics should not be in common ConnectorConfig definiti

[jira] [Updated] (KAFKA-3671) Topics should not be in common ConnectorConfig definitions

2016-05-06 Thread Liquan Pei (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liquan Pei updated KAFKA-3671: -- Summary: Topics should not be in common ConnectorConfig definitions (was: topics should not be in

[jira] [Created] (KAFKA-3671) topics should not be in common ConnectorConfig definitions

2016-05-06 Thread Liquan Pei (JIRA)
Liquan Pei created KAFKA-3671: - Summary: topics should not be in common ConnectorConfig definitions Key: KAFKA-3671 URL: https://issues.apache.org/jira/browse/KAFKA-3671 Project: Kafka Issue

[jira] [Assigned] (KAFKA-3487) Support per-connector/per-task classloaders in Connect

2016-05-05 Thread Liquan Pei (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liquan Pei reassigned KAFKA-3487: - Assignee: Liquan Pei (was: Ewen Cheslack-Postava) > Support per-connector/per-task classload

[jira] [Commented] (KAFKA-3587) LogCleaner fails due to incorrect offset map computation on a replica

2016-05-05 Thread Liquan Pei (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15273584#comment-15273584 ] Liquan Pei commented on KAFKA-3587: --- [~ecomar] [~alekar] Moving the pessimistic c

[jira] [Commented] (KAFKA-3587) LogCleaner fails due to incorrect offset map computation on a replica

2016-05-05 Thread Liquan Pei (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15273305#comment-15273305 ] Liquan Pei commented on KAFKA-3587: --- Hi [~ecomar] I have a WIP PR, mind to take a

[jira] [Updated] (KAFKA-3656) Avoid stressing system more when already under stress

2016-05-04 Thread Liquan Pei (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liquan Pei updated KAFKA-3656: -- Status: Patch Available (was: In Progress) > Avoid stressing system more when already under str

[jira] [Work started] (KAFKA-3656) Avoid stressing system more when already under stress

2016-05-04 Thread Liquan Pei (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on KAFKA-3656 started by Liquan Pei. - > Avoid stressing system more when already under str

[jira] [Assigned] (KAFKA-3649) Add capability to query broker process for configuration properties

2016-05-04 Thread Liquan Pei (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liquan Pei reassigned KAFKA-3649: - Assignee: Liquan Pei > Add capability to query broker process for configuration propert

[jira] [Assigned] (KAFKA-3656) Avoid stressing system more when already under stress

2016-05-04 Thread Liquan Pei (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liquan Pei reassigned KAFKA-3656: - Assignee: Liquan Pei > Avoid stressing system more when already under str

[jira] [Updated] (KAFKA-3654) Config validation should validate both common and connector specific configurations

2016-05-03 Thread Liquan Pei (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liquan Pei updated KAFKA-3654: -- Summary: Config validation should validate both common and connector specific configurations (was

[jira] [Updated] (KAFKA-3654) ConnectorConfig short circuit implementation config validation

2016-05-03 Thread Liquan Pei (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liquan Pei updated KAFKA-3654: -- Description: Right now if you call the validate endpoint with a config that has an invalid value for

[jira] [Updated] (KAFKA-3654) ConnectorConfig short circuit implementation config validation

2016-05-03 Thread Liquan Pei (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liquan Pei updated KAFKA-3654: -- Summary: ConnectorConfig short circuit implementation config validation (was: ConnectorConfig defs

[jira] [Created] (KAFKA-3654) ConnectorConfig defs short circuit implementation config checks

2016-05-03 Thread Liquan Pei (JIRA)
Liquan Pei created KAFKA-3654: - Summary: ConnectorConfig defs short circuit implementation config checks Key: KAFKA-3654 URL: https://issues.apache.org/jira/browse/KAFKA-3654 Project: Kafka

[jira] [Closed] (KAFKA-3611) Remove WARNs when using reflections

2016-04-29 Thread Liquan Pei (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liquan Pei closed KAFKA-3611. - > Remove WARNs when using reflections > > >

[jira] [Closed] (KAFKA-3578) Allow cross origin HTTP requests on all HTTP methods

2016-04-29 Thread Liquan Pei (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liquan Pei closed KAFKA-3578. - > Allow cross origin HTTP requests on all HTTP meth

[jira] [Closed] (KAFKA-3606) Traverse CLASSPATH during herder start to list connectors

2016-04-29 Thread Liquan Pei (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liquan Pei closed KAFKA-3606. - > Traverse CLASSPATH during herder start to list connect

[jira] [Closed] (KAFKA-3582) remove references to Copcyat from connect property files

2016-04-29 Thread Liquan Pei (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liquan Pei closed KAFKA-3582. - > remove references to Copcyat from connect property fi

[jira] [Closed] (KAFKA-3615) Exclude test jars in CLASSPATH of kafka-run-class.sh

2016-04-29 Thread Liquan Pei (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liquan Pei closed KAFKA-3615. - > Exclude test jars in CLASSPATH of kafka-run-class

[jira] [Assigned] (KAFKA-3213) [CONNECT] It looks like we are not backing off properly when reconfiguring tasks

2016-04-29 Thread Liquan Pei (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liquan Pei reassigned KAFKA-3213: - Assignee: Liquan Pei (was: Ewen Cheslack-Postava) > [CONNECT] It looks like we are not back

[jira] [Assigned] (KAFKA-3637) Add method that checks if streams are initialised

2016-04-29 Thread Liquan Pei (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liquan Pei reassigned KAFKA-3637: - Assignee: Liquan Pei > Add method that checks if streams are initiali

[jira] [Assigned] (KAFKA-3627) New consumer doesn't run delayed tasks while under load

2016-04-26 Thread Liquan Pei (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liquan Pei reassigned KAFKA-3627: - Assignee: Liquan Pei (was: Neha Narkhede) > New consumer doesn't run delayed tasks whi

[jira] [Assigned] (KAFKA-3626) Transient failure in testGetAllTopicMetadata

2016-04-26 Thread Liquan Pei (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liquan Pei reassigned KAFKA-3626: - Assignee: Liquan Pei > Transient failure in testGetAllTopicMetad

Re: Kafka missing from ASF Jira?

2016-04-25 Thread Liquan Pei
; Jira, Kafka is not visible in the list of projects in the first field > of the ticket. I see the hundred+ other Apache projects, but no Kafka > :( > -- Liquan Pei Software Engineer, Confluent Inc

[jira] [Commented] (KAFKA-3615) Exclude test jars in CLASSPATH of kafka-run-class.sh

2016-04-24 Thread Liquan Pei (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15255889#comment-15255889 ] Liquan Pei commented on KAFKA-3615: --- [~enothereska] Kafka Connect provides a REST

[jira] [Created] (KAFKA-3615) Exclude test jars in CLASSPATH of kafka-run-class.sh

2016-04-23 Thread Liquan Pei (JIRA)
Liquan Pei created KAFKA-3615: - Summary: Exclude test jars in CLASSPATH of kafka-run-class.sh Key: KAFKA-3615 URL: https://issues.apache.org/jira/browse/KAFKA-3615 Project: Kafka Issue Type

[jira] [Work started] (KAFKA-3611) Remove WARNs when using reflections

2016-04-22 Thread Liquan Pei (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on KAFKA-3611 started by Liquan Pei. - > Remove WARNs when using reflecti

[jira] [Work started] (KAFKA-3611) Remove WARNs when using reflections

2016-04-22 Thread Liquan Pei (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on KAFKA-3611 started by Liquan Pei. - > Remove WARNs when using reflecti

[jira] [Updated] (KAFKA-3611) Remove WARNs when using reflections

2016-04-22 Thread Liquan Pei (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liquan Pei updated KAFKA-3611: -- Status: Patch Available (was: In Progress) > Remove WARNs when using reflecti

[jira] [Work stopped] (KAFKA-3611) Remove WARNs when using reflections

2016-04-22 Thread Liquan Pei (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on KAFKA-3611 stopped by Liquan Pei. - > Remove WARNs when using reflecti

[jira] [Created] (KAFKA-3611) Remove WARNs when using reflections

2016-04-22 Thread Liquan Pei (JIRA)
Liquan Pei created KAFKA-3611: - Summary: Remove WARNs when using reflections Key: KAFKA-3611 URL: https://issues.apache.org/jira/browse/KAFKA-3611 Project: Kafka Issue Type: Improvement

[jira] [Assigned] (KAFKA-3556) Improve group coordinator metrics

2016-04-21 Thread Liquan Pei (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liquan Pei reassigned KAFKA-3556: - Assignee: Liquan Pei > Improve group coordinator metr

[jira] [Resolved] (KAFKA-2479) Add CopycatExceptions to indicate transient and permanent errors in a connector/task

2016-04-21 Thread Liquan Pei (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liquan Pei resolved KAFKA-2479. --- Resolution: Fixed > Add CopycatExceptions to indicate transient and permanent errors i

[jira] [Assigned] (KAFKA-3527) Consumer commitAsync should not expose internal exceptions

2016-04-21 Thread Liquan Pei (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liquan Pei reassigned KAFKA-3527: - Assignee: Liquan Pei (was: Jason Gustafson) > Consumer commitAsync should not expose inter

[jira] [Assigned] (KAFKA-3457) KafkaConsumer.committed(...) hangs forever if port number is wrong

2016-04-21 Thread Liquan Pei (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liquan Pei reassigned KAFKA-3457: - Assignee: Liquan Pei > KafkaConsumer.committed(...) hangs forever if port number is wr

[jira] [Reopened] (KAFKA-3605) Connector REST endpoint allows incorrectly overriding the connector name

2016-04-21 Thread Liquan Pei (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liquan Pei reopened KAFKA-3605: --- > Connector REST endpoint allows incorrectly overriding the connector n

[jira] [Resolved] (KAFKA-3605) Connector REST endpoint allows incorrectly overriding the connector name

2016-04-21 Thread Liquan Pei (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liquan Pei resolved KAFKA-3605. --- Resolution: Fixed > Connector REST endpoint allows incorrectly overriding the connector n

[jira] [Created] (KAFKA-3606) Traverse CLASSPATH during herder start to list connectors

2016-04-21 Thread Liquan Pei (JIRA)
Liquan Pei created KAFKA-3606: - Summary: Traverse CLASSPATH during herder start to list connectors Key: KAFKA-3606 URL: https://issues.apache.org/jira/browse/KAFKA-3606 Project: Kafka Issue Type

[VOTE] KIP-56 Allow cross origin HTTP requests on all HTTP methods

2016-04-21 Thread Liquan Pei
Hi I would like to start vote on KIP-56. https://cwiki.apache.org/confluence/display/KAFKA/KIP-56%3A+Allow+cross+origin+HTTP+requests+on+all+HTTP+methods Thanks, -- Liquan Pei Software Engineer, Confluent Inc

[jira] [Updated] (KAFKA-3459) Returning zero task configurations from a connector does not properly clean up existing tasks

2016-04-20 Thread Liquan Pei (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liquan Pei updated KAFKA-3459: -- Status: Patch Available (was: In Progress) > Returning zero task configurations from a connector d

[jira] [Work started] (KAFKA-3459) Returning zero task configurations from a connector does not properly clean up existing tasks

2016-04-20 Thread Liquan Pei (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on KAFKA-3459 started by Liquan Pei. - > Returning zero task configurations from a connector does not properly clean &

[jira] [Assigned] (KAFKA-3035) Transient: kafka.api.PlaintextConsumerTest > testAutoOffsetReset FAILED

2016-04-20 Thread Liquan Pei (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liquan Pei reassigned KAFKA-3035: - Assignee: Liquan Pei > Transient: kafka.api.PlaintextConsumerTest > testAutoOffsetReset

[jira] [Assigned] (KAFKA-3459) Returning zero task configurations from a connector does not properly clean up existing tasks

2016-04-20 Thread Liquan Pei (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liquan Pei reassigned KAFKA-3459: - Assignee: Liquan Pei (was: Ewen Cheslack-Postava) > Returning zero task configurations fro

[jira] [Updated] (KAFKA-3582) remove references to Copcyat from connect property files

2016-04-18 Thread Liquan Pei (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liquan Pei updated KAFKA-3582: -- Status: Patch Available (was: In Progress) > remove references to Copcyat from connect property fi

[jira] [Work started] (KAFKA-3582) remove references to Copcyat from connect property files

2016-04-18 Thread Liquan Pei (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on KAFKA-3582 started by Liquan Pei. - > remove references to Copcyat from connect property fi

[jira] [Assigned] (KAFKA-3582) remove references to Copcyat from connect property files

2016-04-18 Thread Liquan Pei (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liquan Pei reassigned KAFKA-3582: - Assignee: Liquan Pei > remove references to Copcyat from connect property fi

[jira] [Created] (KAFKA-3583) Docs on pause/resume/restart APIs.

2016-04-18 Thread Liquan Pei (JIRA)
Liquan Pei created KAFKA-3583: - Summary: Docs on pause/resume/restart APIs. Key: KAFKA-3583 URL: https://issues.apache.org/jira/browse/KAFKA-3583 Project: Kafka Issue Type: Improvement

[DISCUSS] KIP-56 Allow cross origin HTTP requests on all HTTP methods

2016-04-18 Thread Liquan Pei
Hi I would like to start a quick discussion on KIP-56 https://cwiki.apache.org/confluence/display/KAFKA/KIP-56%3A+Allow+cross+origin+HTTP+requests+on+all+HTTP+methods This proposal is to allow cross origin HTTP requests on all HTTP methods. Thanks, Liquan -- Liquan Pei Software Engineer

[jira] [Assigned] (KAFKA-3578) Allow cross origin HTTP requests on all HTTP methods

2016-04-18 Thread Liquan Pei (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liquan Pei reassigned KAFKA-3578: - Assignee: Liquan Pei (was: Ewen Cheslack-Postava) > Allow cross origin HTTP requests on

[jira] [Created] (KAFKA-3578) Allow cross origin HTTP requests on all HTTP methods

2016-04-18 Thread Liquan Pei (JIRA)
Liquan Pei created KAFKA-3578: - Summary: Allow cross origin HTTP requests on all HTTP methods Key: KAFKA-3578 URL: https://issues.apache.org/jira/browse/KAFKA-3578 Project: Kafka Issue Type

  1   2   >