Re: [DISCUSS] KIP-504 - Add new Java Authorizer Interface

2019-09-04 Thread Don Bosco Durai
Hi Rajini Overall, it looks good. Thanks Bosco On 9/4/19, 3:39 AM, "Rajini Sivaram" wrote: Hi Colin & Don, I have submitted a PR to help with reviewing the new API: https://github.com/apache/kafka/pull/7293/files. - Authorizer.java contains the updated API and

Jenkins build is back to normal : kafka-trunk-jdk11 #793

2019-09-04 Thread Apache Jenkins Server
See

Re: [DISCUSS] KIP-441: Smooth Scaling Out for Kafka Streams

2019-09-04 Thread John Roesler
I see; thanks for the clarification, Guozhang. If the prior owner of a non-logged, stateful task always reports 0, and all other instances report nothing, then it seems like we need a special case in the assignor to handle assigning these. I.e., the strategy of creating "movement" standbys and wai

[jira] [Created] (KAFKA-8876) KafkaBasedLog does not throw exception when some partitions of the topic is offline

2019-09-04 Thread Boquan Tang (Jira)
Boquan Tang created KAFKA-8876: -- Summary: KafkaBasedLog does not throw exception when some partitions of the topic is offline Key: KAFKA-8876 URL: https://issues.apache.org/jira/browse/KAFKA-8876 Project

Build failed in Jenkins: kafka-trunk-jdk8 #3886

2019-09-04 Thread Apache Jenkins Server
See Changes: [bbejeck] MINOR: remove unnecessary nulllity check (#7282) [bill] KAFKA-8861 Fix flaky -- [...truncated 7.74 MB...] kafka.admin.DeleteConsumerGroupsTest > te

[jira] [Created] (KAFKA-8875) CreateTopic API should check topic existence before replication factor

2019-09-04 Thread Jason Gustafson (Jira)
Jason Gustafson created KAFKA-8875: -- Summary: CreateTopic API should check topic existence before replication factor Key: KAFKA-8875 URL: https://issues.apache.org/jira/browse/KAFKA-8875 Project: Kaf

[jira] [Created] (KAFKA-8874) KIP-517: Add consumer metric indicating time between poll calls

2019-09-04 Thread Kevin Lu (Jira)
Kevin Lu created KAFKA-8874: --- Summary: KIP-517: Add consumer metric indicating time between poll calls Key: KAFKA-8874 URL: https://issues.apache.org/jira/browse/KAFKA-8874 Project: Kafka Issue Ty

Re: [DISCUSS] KIP-450: Sliding Window Aggregations in the DSL

2019-09-04 Thread Guozhang Wang
Hi folks, I've been thinking more about this KIP and my understanding is that we want to introduce a new SlidingWindow notion for aggregation since our current TimeWindow aggregation is not very efficient with very small steps. So I'm wondering that rather than introducing a new implementation mec

[jira] [Created] (KAFKA-8873) Implement timeout for Alter/List PartitionReassignment APIs

2019-09-04 Thread Stanislav Kozlovski (Jira)
Stanislav Kozlovski created KAFKA-8873: -- Summary: Implement timeout for Alter/List PartitionReassignment APIs Key: KAFKA-8873 URL: https://issues.apache.org/jira/browse/KAFKA-8873 Project: Kafka

[jira] [Resolved] (KAFKA-8568) MirrorMaker 2.0 resource leak

2019-09-04 Thread Ryanne Dolan (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-8568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryanne Dolan resolved KAFKA-8568. - Resolution: Fixed > MirrorMaker 2.0 resource leak > - > >

Re: [DISCUSS] KIP-441: Smooth Scaling Out for Kafka Streams

2019-09-04 Thread Guozhang Wang
Hi John, I think I may mis-communicate my ideas a bit, what I meant is simply for case 1), let the active tasks reporting a lag of 0 instead of not reporting any lags at all; in addition, for other hosts not hosting this tasks, and hence we do not know the offset lag at all we set the "StatefulTas

Jenkins build is back to normal : kafka-2.2-jdk8 #163

2019-09-04 Thread Apache Jenkins Server
See

[jira] [Reopened] (KAFKA-7940) Flaky Test CustomQuotaCallbackTest#testCustomQuotaCallback

2019-09-04 Thread Matthias J. Sax (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-7940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax reopened KAFKA-7940: Failed again on `trunk`  [https://builds.apache.org/blue/organizations/jenkins/kafka-trunk-jdk1

[jira] [Resolved] (KAFKA-8460) Flaky Test PlaintextConsumerTest#testLowMaxFetchSizeForRequestAndPartition

2019-09-04 Thread Matthias J. Sax (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-8460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax resolved KAFKA-8460. Resolution: Duplicate > Flaky Test PlaintextConsumerTest#testLowMaxFetchSizeForRequestAndP

Re: [DISCUSS] KIP-441: Smooth Scaling Out for Kafka Streams

2019-09-04 Thread John Roesler
Thanks for the reply, Guozhang, I'll start by re-stating your cases, just to make sure we're on the same page... 1) The task is stateful, and all its stores are non-logged. For this case under the proposal, we would not have any standbys and the active version would actually not report any lag at

[jira] [Reopened] (KAFKA-8677) Flakey test GroupEndToEndAuthorizationTest#testNoDescribeProduceOrConsumeWithoutTopicDescribeAcl

2019-09-04 Thread Matthias J. Sax (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-8677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax reopened KAFKA-8677: This test is still failing:  [https://builds.apache.org/blue/organizations/jenkins/kafka-trunk-

Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and TrustStore

2019-09-04 Thread Maulin Vasavada
Do you guys think it would be easier if you can provide comments on GitHub and we can continue there and summarize the conclusion here? We should not lose addressing any comments. On Wed, Sep 4, 2019 at 12:34 PM Pellerin, Clement wrote: > The proposed interface does not look like the Builder pa

Re: [VOTE] KIP-482: The Kafka Protocol should Support Optional Tagged Fields

2019-09-04 Thread Colin McCabe
On Wed, Sep 4, 2019, at 13:01, Jason Gustafson wrote: > Hi Colin, > > Just a couple questions. > > 1. I think we discussed that we would do a lazy version bump of all > protocols in order to get flexible version support. Can you add that to the > KIP? Good point. Added. > 2. The doc mentions a

Re: [VOTE] KIP-482: The Kafka Protocol should Support Optional Tagged Fields

2019-09-04 Thread Jason Gustafson
Hi Colin, Just a couple questions. 1. I think we discussed that we would do a lazy version bump of all protocols in order to get flexible version support. Can you add that to the KIP? 2. The doc mentions a bump to the request and response header formats to version 1. Currently there is no formal

Build failed in Jenkins: kafka-trunk-jdk11 #792

2019-09-04 Thread Apache Jenkins Server
See Changes: [bbejeck] MINOR: remove unnecessary nulllity check (#7282) [bill] KAFKA-8861 Fix flaky -- [...truncated 2.62 MB...] org.apache.kafka.connect.integration.Reba

RE: [DISCUSS] KIP-486 Support for pluggable KeyStore and TrustStore

2019-09-04 Thread Pellerin, Clement
The proposed interface does not look like the Builder pattern I am used to. Should SslEngineBuilder be called SslEngineFactory instead? On Mon, Sep 2, 2019, at 03:33, Rajini Sivaram wrote: > I would expect SslEngineBuilder interface to look something like this, > perhaps with some tweaking: > > p

Re: [DISCUSS] KIP-444: Augment Metrics for Kafka Streams

2019-09-04 Thread Bruno Cadonna
Hi, I am sorry to restart the discussion here, but I came across a small issue in the KIP. I started to implement KIP-444 and I am bit concerned about the values for the the config `built.in.metrics.version`. In the KIP the possible values are specified as all Kafka Streams versions. I think that

Re: [DISCUSS] KIP-441: Smooth Scaling Out for Kafka Streams

2019-09-04 Thread Guozhang Wang
Hello John, Your reasoning about non-logged state store looks good to me. But I think it worth considering two cases differently: 1) a task's all state stores are non-logged, 2) a task has non-logged state store but also have other logged state store. For 1), we should not have any standbys for s

Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and TrustStore

2019-09-04 Thread Maulin Vasavada
Hi Colin Thanks for your comments. I agree with most of them. This is not a pull-request ready code yet :) If we want to make DefaultSslEngineBuilder final then what do you propose to address our requirements to be able to plugin custom way for loading keys/certs? (The original challenge we have

Re: [DISCUSS] KIP-441: Smooth Scaling Out for Kafka Streams

2019-09-04 Thread Bruno Cadonna
Hi John, Thank you for your answer. Your assumptions sound reasonable to me. Best, Bruno On Wed, Sep 4, 2019 at 5:30 PM John Roesler wrote: > > Hey Bruno, > > Thanks for taking another look. Some quick responses: > > 1) It just means the number of offsets in the topic. E.g., the LSO is 100, > b

[DISCUSS] KIP-516: Topic Identifiers

2019-09-04 Thread Lucas Bradstreet
Hi all, I would like to kick off discussion of KIP-516, an implementation of topic IDs for Kafka. Topic IDs aim to solve topic uniqueness problems in Kafka, where referring to a topic by name alone is insufficient. Such cases include when a topic has been deleted and recreated with the same name.

[jira] [Created] (KAFKA-8872) Improvements to controller "deleting" state / topic Identifiers

2019-09-04 Thread Lucas Bradstreet (Jira)
Lucas Bradstreet created KAFKA-8872: --- Summary: Improvements to controller "deleting" state / topic Identifiers Key: KAFKA-8872 URL: https://issues.apache.org/jira/browse/KAFKA-8872 Project: Kafka

Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and TrustStore

2019-09-04 Thread Colin McCabe
On Tue, Sep 3, 2019, at 22:56, Maulin Vasavada wrote: > Hi all > > Please check > https://github.com/maulin-vasavada/kafka/commit/44f86395b1ba3fe4bd87de89029d72da77995ff8 > > > This is just the first cut obviously. There are few call outs I would like > to make, > > 1. So far I kept the old Ssl

[VOTE] KIP-447: Producer scalability for exactly once semantics

2019-09-04 Thread Boyang Chen
Hey all, I would like to start the vote for KIP-447 . This is a very important step to improve Kafka Streams scalability in exactly-once semantics, by avoiding linearly increasing number o

Build failed in Jenkins: kafka-2.1-jdk8 #227

2019-09-04 Thread Apache Jenkins Server
See Changes: [bill] KAFKA-8861 Fix flaky [bill] HOTFIX: The cherry-pick for https://github.com/apache/kafka/pull/7207 to -- Started by an SCM change Started by an SCM change

Re: [DISCUSS] KIP-447: Producer scalability for exactly once semantics

2019-09-04 Thread Boyang Chen
>From offline discussion, the eventual conclusion is to use a top-level Consumer#getMetadata() API to fetch the latest group metadata information for offset fencing, so that we only call initTransaction once in lifetime. Since no further question is raised on this thread, I will start vote today.

[jira] [Created] (KAFKA-8871) Allow timestamp manipulation in ValueTransformerWithKey

2019-09-04 Thread Levani Kokhreidze (Jira)
Levani Kokhreidze created KAFKA-8871: Summary: Allow timestamp manipulation in ValueTransformerWithKey Key: KAFKA-8871 URL: https://issues.apache.org/jira/browse/KAFKA-8871 Project: Kafka

Build failed in Jenkins: kafka-2.0-jdk8 #291

2019-09-04 Thread Apache Jenkins Server
See Changes: [bill] KAFKA-8861 Fix flaky -- Started by an SCM change Started by an SCM change [EnvInject] - Loading node environment variables. Building remotely on H24 (ubun

[jira] [Created] (KAFKA-8870) Prevent dirty reads of Streams state store from Interactive queries

2019-09-04 Thread Vinoth Chandar (Jira)
Vinoth Chandar created KAFKA-8870: - Summary: Prevent dirty reads of Streams state store from Interactive queries Key: KAFKA-8870 URL: https://issues.apache.org/jira/browse/KAFKA-8870 Project: Kafka

Build failed in Jenkins: kafka-2.3-jdk8 #100

2019-09-04 Thread Apache Jenkins Server
See Changes: [bill] KAFKA-8861 Fix flaky -- Started by an SCM change Started by an SCM change Started by an SCM change [EnvInject] - Loading node environment variables. Build

[jira] [Created] (KAFKA-8869) Map taskConfigs in KafkaConfigBackingStore grows monotonically despite of task removals

2019-09-04 Thread Konstantine Karantasis (Jira)
Konstantine Karantasis created KAFKA-8869: - Summary: Map taskConfigs in KafkaConfigBackingStore grows monotonically despite of task removals Key: KAFKA-8869 URL: https://issues.apache.org/jira/browse/KAFKA

[jira] [Created] (KAFKA-8868) Consider auto-generating Streams binary protocol messages

2019-09-04 Thread John Roesler (Jira)
John Roesler created KAFKA-8868: --- Summary: Consider auto-generating Streams binary protocol messages Key: KAFKA-8868 URL: https://issues.apache.org/jira/browse/KAFKA-8868 Project: Kafka Issue T

Re: [DISCUSS] KIP-441: Smooth Scaling Out for Kafka Streams

2019-09-04 Thread John Roesler
Hey Bruno, Thanks for taking another look. Some quick responses: 1) It just means the number of offsets in the topic. E.g., the LSO is 100, but the first offset is 40 due to retention, so there are 60 offsets in the topic. Further, the lag on that topic would be considered to be 60 for any task t

Re: [VOTE] KIP-482: The Kafka Protocol should Support Optional Tagged Fields

2019-09-04 Thread David Arthur
+1 binding. Thanks for the KIP, Colin! -David On Wed, Sep 4, 2019 at 5:40 AM Harsha Chintalapani wrote: > LGTM. +1 (binding) > -Harsha > > > On Wed, Sep 04, 2019 at 1:46 AM, Satish Duggana > wrote: > > > +1 (non-binding) Thanks for the nice KIP. > > > > You may want to update the KIP saying t

[jira] [Created] (KAFKA-8867) Kafka Connect JDBC fails to create PostgreSQL table with default boolean value

2019-09-04 Thread Tudor-Alexandru Voicu (Jira)
Tudor-Alexandru Voicu created KAFKA-8867: Summary: Kafka Connect JDBC fails to create PostgreSQL table with default boolean value Key: KAFKA-8867 URL: https://issues.apache.org/jira/browse/KAFKA-8867

Build failed in Jenkins: kafka-trunk-jdk8 #3885

2019-09-04 Thread Apache Jenkins Server
See Changes: [github] MINOR: Add system configuration to zk security exception messages [matthias] KAFKA-7849: Fix the warning when using GlobalKTable (#7104) -- [...trun

Re: Request to be added as a contributor

2019-09-04 Thread Bill Bejeck
Soontaek, You're all set now. Thanks for your interest in Apache Kafka. -Bill On Wed, Sep 4, 2019 at 10:08 AM Soontaek Lim wrote: > Hi Team, > > Request you to add me to the contributor list. > Jira username: soontaek.lim > > Thank you > > Best regards, > Soontaek Lim >

Request to be added as a contributor

2019-09-04 Thread Soontaek Lim
Hi Team, Request you to add me to the contributor list. Jira username: soontaek.lim Thank you Best regards, Soontaek Lim

[jira] [Resolved] (KAFKA-8676) Avoid Stopping Unnecessary Connectors and Tasks

2019-09-04 Thread Randall Hauch (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-8676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Randall Hauch resolved KAFKA-8676. -- Fix Version/s: (was: 2.3.0) 2.3.1 2.4.0 Revie

Build failed in Jenkins: kafka-trunk-jdk11 #791

2019-09-04 Thread Apache Jenkins Server
See Changes: [github] MINOR: Add system configuration to zk security exception messages [matthias] KAFKA-7849: Fix the warning when using GlobalKTable (#7104) -- [...trun

Re: [DISCUSS] KIP-504 - Add new Java Authorizer Interface

2019-09-04 Thread Ismael Juma
Hi Jun, I think it's important to discuss the pluggable API versus the calling implementation as two separate points. >From an API perspective, are you suggesting that we should tell users that they cannot block in authorize? Or are you saying that it's OK to block on authorize on occasion and th

Re: [DISCUSS] KIP-504 - Add new Java Authorizer Interface

2019-09-04 Thread Jun Rao
Hi, Rajini, Thanks for the KIP. I was concerned about #4 too. If we change the handling of all requests to use an async authorize() api, will that cause the code much harder to understand? There are quite a few callbacks already. I am not sure that we want to introduce more of those. The benefit f

[jira] [Created] (KAFKA-8866) Make Authorizer create/delete exceptions Optional

2019-09-04 Thread Rajini Sivaram (Jira)
Rajini Sivaram created KAFKA-8866: - Summary: Make Authorizer create/delete exceptions Optional Key: KAFKA-8866 URL: https://issues.apache.org/jira/browse/KAFKA-8866 Project: Kafka Issue Type:

[jira] [Resolved] (KAFKA-8760) KIP-504: Add new Java Authorizer API

2019-09-04 Thread Rajini Sivaram (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-8760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajini Sivaram resolved KAFKA-8760. --- Resolution: Fixed > KIP-504: Add new Java Authorizer API > -

[jira] [Created] (KAFKA-8865) KIP-504: New Java Authorizer API

2019-09-04 Thread Rajini Sivaram (Jira)
Rajini Sivaram created KAFKA-8865: - Summary: KIP-504: New Java Authorizer API Key: KAFKA-8865 URL: https://issues.apache.org/jira/browse/KAFKA-8865 Project: Kafka Issue Type: Improvement

Re: [DISCUSS] KIP-441: Smooth Scaling Out for Kafka Streams

2019-09-04 Thread Bruno Cadonna
Hi, 1) What do you mean with "full set of offsets in the topic"? Is this the sum of all offsets of the changelog partitions of the task? 2) I am not sure whether non-logged stateful tasks should be effectively treated as stateless tasks during assignment. First we need to decide whether a non-log

Re: [DISCUSS] KIP-504 - Add new Java Authorizer Interface

2019-09-04 Thread Rajini Sivaram
Hi Colin & Don, I have submitted a PR to help with reviewing the new API: https://github.com/apache/kafka/pull/7293/files. - Authorizer.java contains the updated API and includes javadoc update including the threading model. - AclAuthorizer.scala changes in the PR should give an idea of

Current master does not build

2019-09-04 Thread Florian Schmidt
Hello, I'm trying to compile the current master (commit: 8dc80e22973d89090833cb791d9269e9a0598059), but the build fails at the tasks :core:compileScala and :kafka:core:compileScala with the following error: .../kafka/core/build/classes/scala/main/kafka/api/package.class' is broken (class java.lan

Re: [VOTE] KIP-482: The Kafka Protocol should Support Optional Tagged Fields

2019-09-04 Thread Harsha Chintalapani
LGTM. +1 (binding) -Harsha On Wed, Sep 04, 2019 at 1:46 AM, Satish Duggana wrote: > +1 (non-binding) Thanks for the nice KIP. > > You may want to update the KIP saying that optional tagged fields do not > support complex types(or structs). > > On Wed, Sep 4, 2019 at 3:43 AM Jose Armando Garcia

Kafka to Kudu

2019-09-04 Thread Sarfraz SMS
Hello, Greetings people, I want to move the data from kafka brokers consumed into Kudu cloud database. Is there any pre requisites needed in doing this ? Any suggestions Do's/Don'ts in implementing this ? Thanks, Sarfraz

Re: [VOTE] KIP-482: The Kafka Protocol should Support Optional Tagged Fields

2019-09-04 Thread Satish Duggana
+1 (non-binding) Thanks for the nice KIP. You may want to update the KIP saying that optional tagged fields do not support complex types(or structs). On Wed, Sep 4, 2019 at 3:43 AM Jose Armando Garcia Sancio wrote: > > +1 (non-binding) > > Looking forward to this improvement. > > On Tue, Sep 3,

Re: [DISCUSS] KIP-504 - Add new Java Authorizer Interface

2019-09-04 Thread Rajini Sivaram
Hi Don, Thanks for your note. 1) The intention is to avoid blocking in the calling thread. We already have several requests that are put into a purgatory when waiting for remote communication, for example produce request waiting for replication. Since we have a limited number of request threads i

Build failed in Jenkins: kafka-trunk-jdk11 #790

2019-09-04 Thread Apache Jenkins Server
See Changes: [rhauch] Changed for updatedTasks, avoids stopping and starting of unnecessary [rhauch] MINOR: Add unit test for KAFKA-8676 to guard against unrequired task

Re: [VOTE] KIP-496: Administrative API to delete consumer offsets

2019-09-04 Thread David Jacot
Hi all, While implementing the KIP, I have realized that a new error code and exception is required to notify the caller that offsets of a topic can not be deleted because the group is actively subscribed to the topic. I would like to know if there are any concerns with these changes before updat