Re: [DISCUSS] Apache Kafka 2.4.0 release

Hi Manikumar, Could we add KIP-511 to the plan? I think it will make it. Thanks, David On Tue, Aug 27, 2019 at 5:32 PM Manikumar wrote: > Hi all, > > I put together a draft release plan with Oct 2019 as the release month and > a list of KIPs that have already been voted: > > https://cwiki.apac

[jira] [Created] (KAFKA-8881) Measure thread running time precisely

huxihx created KAFKA-8881: - Summary: Measure thread running time precisely Key: KAFKA-8881 URL: https://issues.apache.org/jira/browse/KAFKA-8881 Project: Kafka Issue Type: Improvement Affects Ver

Re: [VOTE] KIP-441: Smooth Scaling Out for Kafka Streams

+1 (non-binding) On Fri, Sep 6, 2019 at 12:32 AM Guozhang Wang wrote: > > +1 (binding). > > On Thu, Sep 5, 2019 at 2:47 PM John Roesler wrote: > > > Hello, all, > > > > After a great discussion, I'd like to open voting on KIP-441, > > to avoid long restore times in Streams after rebalancing. > >

Re: [DISCUSS] Apache Kafka 2.4.0 release

Hi David, Yes, we can include KIP-511. KIP must be accepted by KIP Freeze date (Sep 25, 2019 ) Thanks, On Fri, Sep 6, 2019 at 12:53 PM David Jacot wrote: > Hi Manikumar, > > Could we add KIP-511 to the plan? I think it will make it. > > Thanks, > David > > On Tue, Aug 27, 2019 at 5:32 PM Man

[jira] [Created] (KAFKA-8882) It's not possible to restart Kafka Streams using StateListener

Jakub created KAFKA-8882: Summary: It's not possible to restart Kafka Streams using StateListener Key: KAFKA-8882 URL: https://issues.apache.org/jira/browse/KAFKA-8882 Project: Kafka Issue Type: Bug

[jira] [Created] (KAFKA-8883) Kafka Consumer continuously displaying info message Seeking to offset 0 for partition

Veerabhadra Rao Mallavarapu created KAFKA-8883: -- Summary: Kafka Consumer continuously displaying info message Seeking to offset 0 for partition Key: KAFKA-8883 URL: https://issues.apache.org/jira/brow

[jira] [Created] (KAFKA-8884) Improve streams errors on class cast exception in ProcessorsNodes

Antony Stubbs created KAFKA-8884: Summary: Improve streams errors on class cast exception in ProcessorsNodes Key: KAFKA-8884 URL: https://issues.apache.org/jira/browse/KAFKA-8884 Project: Kafka

Build failed in Jenkins: kafka-trunk-jdk11 #798

See Changes: [jason] KAFKA-8822; Consolidate `PartitionRecords` and `CompletedFetch` types in [jason] KAFKA-8590; Use automated TxnOffsetCommit type and add tests for --

Help needed with PR #6592

Hi, I’ve been maintaining this PR "KAFKA-8326: Introduce List Serde" https://github.com/apache/kafka/pull/6592 My linters keep failing during the building, complaining about: "Disallowed import - org.apache.kafka.clients.CommonClientConfigs.” in my l

RE: [DISCUSS] KIP-486 Support for pluggable KeyStore and TrustStore

This is the way I see it, which is in no way authoritative. First you have to find a complete solution, then you need to document it in a KIP and that KIP needs to pass a vote. Any votes before the KIP vote starts is meaningless. As for the ownership and authorship of the KIPs, I don't plan to

Re: Help needed with PR #6592

Hi Daniyar, You'll need to edit checkstyle/import-control.xml to allow it. Cheers, Tom On Fri, Sep 6, 2019 at 3:40 PM Development wrote: > Hi, > > I’ve been maintaining this PR "KAFKA-8326: Introduce List Serde" > https://github.com/apache/kafka/pull/6592 < > https://github.com/apache/kafka/p

Re: [DISCUSS] 2.3.1 Bug Fix Release

Thanks for volunteering David, +1. On Thu, Sep 5, 2019, 3:50 PM David Arthur wrote: > Hey everyone, > > I'd like to volunteer for the Kafka 2.3.1 bug fix release. Kafka 2.3.0 was > released last month on August 6 and a number of issues have been fixed > since then including several critical and

Re: [DISCUSS] KIP-504 - Add new Java Authorizer Interface

Hi Jun, For ACLs, the size is also dependent on the number of users. So in deployments with large number of users where some users are not active, an LRU cache may be useful. Agree that there may be other solutions that help avoid the requirement for an async API even for this case. I wasn't expe

Re: [DISCUSS] KIP-504 - Add new Java Authorizer Interface

I'm on the move, but what Rajini said seems reasonable. I don't think using SSDs solves the issue. They can still hang for seconds when they fail. Also, many people may not have local SSDs available (remote SSDs like EBS hang for tens of seconds when there are issues). We are currently vulnerable

Re: [DISCUSS] KIP-504 - Add new Java Authorizer Interface

One more thing: if people have strong opinions that authorize has to be sync, we could leave it like that for now. If needed, we can later add an authorizeAsync with another method returning a Boolean indicating that it's supported. That is, there is a path to evolve the interface (even if a bit ug

Re: [DISCUSS] KIP-504 - Add new Java Authorizer Interface

+1 I might be wrong here, but here are few of my assumptions.. 1. Unlike transactional services, in Kafka most of the users are application users, so once the connection is established, then they are going to be long running sessions. So the caching is more predictable and heavy lifting can be

Re: [DISCUSS] KIP-450: Sliding Window Aggregations in the DSL

Thanks for this idea, Guozhang, it does seem to be a nice way to solve the problem. I'm a _little_ concerned about the interface, though. It might be better to just add a new argument to a new method overload like `(initializer, aggregator, merger/combinator/whatever)`. Two reasons come to mind f

Re: [DISCUSS] KIP-504 - Add new Java Authorizer Interface

I would suggest that we go with synchronous authorize() and acls() method to describe ACLs, but asynchronous createAcls() and deleteAcls() that return CompletionStage. With KIP-500, I think async create/delete methods will be useful. For now, we can assume that brokers are able to cache all ACLs. T

Re: [DISCUSS] KIP-504 - Add new Java Authorizer Interface

Hi Rajini, I agree that there is probably some value in async createAcls and deleteAcls methods. It's also a smaller change to use a futures-based interface there, since it really only affects the CreateAcls and DeleteAcls operations. It's probably fine for the other methods to be sync for no

Re: [DISCUSS] KIP-494 Connect REST Endpoint for Transformations (SMTs) and other Plugins

I've updated the KIP here to simplify and clarify the goal -- it's a pretty simple KIP to add a REST endpoint for SMTs and other plugins. This will make it easier to see what SMTs you've loaded. Hoping for some good discussion! Thanks, all On Sat, Jul 20, 2019 at 9:07 PM Cyrus Vafadari wrote:

_consumer_offsets is becoming rather big. How to purge?

_consumer_offsets is becoming rather big > 1 TB. Is there a way to purge dead/inactive consumer id rows from it? I am assuming dead/inactive consumer id rows is the reason. could there be another reason? [reposting from user list]

How to test the performance of Kafka consumer?

I want to test the throughput and latency of Kafka consumer through the application, but there is no test entry. How to test and collect throughput and latency data?

Re: [DISCUSS] KIP-491: Preferred Leader Deprioritized List (Temporary Blacklist)

Hi,  Just want to ping and bubble up the discussion of KIP-491.  On a large scale of Kafka clusters with thousands of brokers in many clusters.  Frequent hardware failures are common, although the reassignments to change the preferred leaders is a workaround, it incurs unnecessary additional w

Jenkins build is back to normal : kafka-trunk-jdk11 #799

See

Re: [DISCUSS] KIP-504 - Add new Java Authorizer Interface

Sounds good to me. Thanks Bosco On 9/6/19, 11:47 AM, "Colin McCabe" wrote: Hi Rajini, I agree that there is probably some value in async createAcls and deleteAcls methods. It's also a smaller change to use a futures-based interface there, since it really only affects the Crea

Re: [DISCUSS] KIP-504 - Add new Java Authorizer Interface

It's not uncommon for people to do large numbers of admin operations per second. We've seen it many times in production. It's often a bug, but the system must cope with it without affecting other users of the same Kafka cluster. Ismael On Fri, Sep 6, 2019, 9:32 AM Don Bosco Durai wrote: > +1 >

Re: [VOTE] KIP-482: The Kafka Protocol should Support Optional Tagged Fields

+1 Thanks Colin. This is really going to help with compatibility. -Jason On Wed, Sep 4, 2019 at 1:34 PM Colin McCabe wrote: > On Wed, Sep 4, 2019, at 13:01, Jason Gustafson wrote: > > Hi Colin, > > > > Just a couple questions. > > > > 1. I think we discussed that we would do a lazy version bump

Re: [DISCUSS] KIP-491: Preferred Leader Deprioritized List (Temporary Blacklist)

One possibility would be writing a new command-line tool that would deprioritize a given replica using the new KIP-455 API. Then it could write out a JSON files containing the old priorities, which could be restored when (or if) we needed to do so. This seems like it might be simpler and easie

Re: [DISCUSS] KIP-507: Securing Internal Connect REST Endpoints

Hi all, I've published a draft PR implementing the changes currently proposed in KIP-507, which can be found at https://github.com/apache/kafka/pull/7310. Thanks for all of the review and feedback so far! Given the lack of any major voiced reservations so far, if I don't hear anything else over th

Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and TrustStore

Hi Maulin, Clement brought up a good point, which is that we should have agreement on the overall design before we start looking at PRs. In Kafka, that means starting a discussion on a KIP, and then getting that KIP voted in. There is more information about the KIP process here: https://cwiki.

Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and TrustStore

Sorry, I meant to write "removing an API typically requires a new major release of Kafka". Deprecating an API can be done in a minor release. regards, Colin On Fri, Sep 6, 2019, at 14:49, Colin McCabe wrote: > Hi Maulin, > > Clement brought up a good point, which is that we should have agreem

Re: [VOTE] KIP-482: The Kafka Protocol should Support Optional Tagged Fields

Hi all, The KIP-482 vote has passed. Thanks to everyone who voted or discussed the issue. There were 3 non-binding +1 votes from David Jacot, Jose Armando Garcia Sancio, and Satish Duggana. There were 3 binding +1 votes from Harsha Chintalapani, David Arthur, and Jason Gustafson. best, Colin

Re: [DISCUSS] KIP-511: Collect and Expose Client's Name and Version in the Brokers

Hi David, Yeah, this is kind of difficult. >From the high level, we either need forward compatibility (old brokers able to >read new ApiVersionsRequests) or some kind of challenge/response system. I >think you're on the right track to be thinking about forward compatibility... >a challenge/r

[jira] [Created] (KAFKA-8885) The Kafka Protocol should Support Optional Tagged Fields

Colin P. McCabe created KAFKA-8885: -- Summary: The Kafka Protocol should Support Optional Tagged Fields Key: KAFKA-8885 URL: https://issues.apache.org/jira/browse/KAFKA-8885 Project: Kafka Is

Re: [DISCUSS] KIP-495: Dynamically Adjust Log Levels in Connect

Hi Arjun, The updated KIP looks good. Just a couple questions: 1. Is the /admin endpoint on the normal listener by default? If not, is there a way to have it use the same listener? 2. Changes to logging configuration are not intended to be persistent, is that right? Also, I assume changes only ap

Re: [DISCUSS] KIP-495: Dynamically Adjust Log Levels in Connect

Thanks a lot, Jason! Answers inline. I'll also modify the kip to make these clear. On Fri, Sep 6, 2019 at 4:01 PM Jason Gustafson wrote: > Hi Arjun, > > The updated KIP looks good. Just a couple questions: > > 1. Is the /admin endpoint on the normal listener by default? If not, is > there a way

Re: [DISCUSS] KIP-491: Preferred Leader Deprioritized List (Temporary Blacklist)

Hi Colin, Thanks for the feedback.  The "separate set of metadata about blacklists" in KIP-491 is just the list of broker ids. Usually 1 or 2 or a couple in the cluster.  Should be easier than keeping json files?  e.g. what if we first blacklist broker_id_1, then another broker_id_2 has issues

Re: [VOTE] KIP-447: Producer scalability for exactly once semantics

Thanks Guozhang, I have polished the design doc to make it sync with current KIP. As for overriding default timeout values, I guess it's already stated in the KIP to set txn timeout to 10s, are you suggesting we should also put down this recommendation on the KIP for non-stream EOS users? Boyang

Build failed in Jenkins: kafka-trunk-jdk8 #3892

See Changes: [github] MINOR: Add cause to thrown exception when deleting topic in TopicCommand -- [...truncated 2.62 MB...] org.apache.kafka.connect.converters.ByteArray

Build failed in Jenkins: kafka-trunk-jdk11 #800

See Changes: [github] MINOR: Add cause to thrown exception when deleting topic in TopicCommand -- [...truncated 2.61 MB...] org.apache.kafka.streams.StreamsConfigTest > t

Re: [DISCUSS] KIP-470: TopologyTestDriver test input and output usability improvements

Hi, Let's get back to this after summer break. Thanks Antony to offering help, it might be needed. I modified the KIP based on the feedback to be a mixture of variations 4 and 5. In TestInputTopic I removed deprecation from one variation with long timestamp and removed totally one version withou

Re: [DISCUSS] KIP-470: TopologyTestDriver test input and output usability improvements

Hi, Sorry; I need to rollback right away the one method removal what I was proposing. One constructor with Long restored to TestRecord, which is needed by TestInputTopic. Regards, Jukka la 7. syysk. 2019 klo 8.06 Jukka Karvanen (jukka.karva...@jukinimi.com) kirjoitti: > Hi, > > Let's get back