Re: [DISCUSS] KIP-567: Kafka Cluster Audit (new discussion)

Hi, Thanks for the KIP. If the auditor needs access to the details of the action, one could argue that even the response should be passed down to the auditor. Is it feasible to convert the Java requests and responses to public API? If not, do we have another option to access this info in the audi

Re: [VOTE] KIP-631: The Quorum-based Kafka Controller

Thanks for the KIP. +1 (non-binding) On Tue, Sep 15, 2020 at 12:23 AM Colin McCabe wrote: > Hi all, > > I'd like to call a vote on KIP-631: the quorum-based Kafka Controller. > The KIP is here: > > https://cwiki.apache.org/confluence/x/4RV4CQ > > The DISCUSS thread is here: > > > https://lists.

Build failed in Jenkins: Kafka » kafka-trunk-jdk8 #69

See Changes: [github] MINOR: Fix common struct `JsonConverter` and `Schema` generation (#9279) -- [...truncated 6.53 MB...] org.apache.kafka.streams.test.Outp

[jira] [Resolved] (KAFKA-10495) Fix spelling mistake

[ https://issues.apache.org/jira/browse/KAFKA-10495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] 欧阳能达 resolved KAFKA-10495. -- Resolution: Won't Fix > Fix spelling mistake > > > Key: KAFKA-10495 >

Build failed in Jenkins: Kafka » kafka-trunk-jdk15 #69

See Changes: [github] MINOR: Fix common struct `JsonConverter` and `Schema` generation (#9279) -- [...truncated 3.29 MB...] org.apache.kafka.streams.test.Ou

Build failed in Jenkins: Kafka » kafka-trunk-jdk11 #70

See Changes: [github] MINOR: Fix common struct `JsonConverter` and `Schema` generation (#9279) -- [...truncated 3.29 MB...] org.apache.kafka.streams.test.Con

Jenkins build is back to normal : Kafka » kafka-trunk-jdk15 #68

See

Re: [DISCUSS] KIP-516: Topic Identifiers

Hi Jun, Thanks for the quick response! 12. I've decided to bump up the versions on the requests and updated the KIP. I think it's good we thoroughly discussed the options here, so we know we made a good choice. :) 13. This is an interesting situation. I think if this does occur we should give a w

Build failed in Jenkins: Kafka » kafka-trunk-jdk8 #68

See Changes: [github] MINOR: Fix now that kafka.apache.org resolves to 3 IP addresses (#9294) -- [...truncated 3.27 MB...] org.apache.kafka.streams.TestTopics

Build failed in Jenkins: Kafka » kafka-trunk-jdk11 #69

See Changes: [github] MINOR: Fix now that kafka.apache.org resolves to 3 IP addresses (#9294) -- [...truncated 3.29 MB...] org.apache.kafka.streams.test.Consu

Re: [DISCUSS] KIP-516: Topic Identifiers

Hi, Justine, Thanks for the reply. 12. I don't have a strong preference either. However, if we need IBP anyway, maybe it's easier to just bump up the version for all inter broker requests and add the topic id field as a regular field. A regular field is a bit more concise in wire transfer than a

Re: [DISCUSS] KIP-631: The Quorum-based Kafka Controller

Hi Unmesh, That's a fair point. I have moved the lease duration to the broker heartbeat response. That way lease durations can be changed just be reconfiguring the controllers. best, Colin On Wed, Sep 16, 2020, at 07:40, Unmesh Joshi wrote: > Thanks Colin, the changes look good to me. One sm

Re: [DISCUSS] KIP-516: Topic Identifiers

Hi Jun, Thanks for the comments. 12. I bumped the LeaderAndIsrRequest because I removed the topic name field in the response. It may be possible to avoid bumping the version without that change, but I may be missing something. I believe StopReplica is actually on version 3 now, but because version

Re: [DISCUSS] KIP-516: Topic Identifiers

Hi, Justine, Thanks for the reply. A couple of more comments below. 12. ListOffset and OffsetForLeader currently don't support flexible fields. So, we have to bump up the version number and use IBP at least for these two requests. Note that it seems 2.7.0 will require IBP anyway because of change

Re: [DISCUSS] KIP-516: Topic Identifiers

Hello all, I've thought some more about removing the topic name field from some of the requests. On closer inspection of the requests/responses, it seems that the internal changes would be much larger than I expected. Some protocols involve clients, so they would require changes too. I'm thinking

[jira] [Created] (KAFKA-10499) 4 Unit Tests are breaking after addition of a new A record to "apache.org"

Prateek Agarwal created KAFKA-10499: --- Summary: 4 Unit Tests are breaking after addition of a new A record to "apache.org" Key: KAFKA-10499 URL: https://issues.apache.org/jira/browse/KAFKA-10499 Proj

Serious Java Consumer performance issues / expectations vs. librdkafka

Hi I am trying to use a plain Java consumer (over SSL) to consume a very large amount of historic data (20+TB across 20+ partitions). Consumption performance is very low when fully parallelized. We are seeing about* 200k rec/s* with java consumer versus *950k rec/s* with librdkafka We are seeing

Re: [DISCUSS] KIP-508: Make Suppression State Queriable - rebooted.

Hi John, I just reviewed KIP-307. As far as I understood, ... 1. There was Materialized name initially. 2. With KIP-307, Named Operations were added. 3. Now we have two options for materializing suppression. If we take Materialized name here, we have two names for the same operation, which is not

Re: [DISCUSS] KIP-664: Provide tooling to detect and abort hanging transactions

Thanks for the updates Jason. I'm pretty satisfied with the overall motivation and proposed solution, just a couple of more comments. 1. Why do we need to use type string for `StatesFilter` instead of a short value, as we could translate it and save space? 2. I'm wondering whether the requirement

[jira] [Created] (KAFKA-10498) Consumer should do offset/epoch validation through `Fetch` when possible

Jason Gustafson created KAFKA-10498: --- Summary: Consumer should do offset/epoch validation through `Fetch` when possible Key: KAFKA-10498 URL: https://issues.apache.org/jira/browse/KAFKA-10498 Projec

[jira] [Created] (KAFKA-10497) Convert group/transaction coordinator metadata schemas to use generated protocol

Jason Gustafson created KAFKA-10497: --- Summary: Convert group/transaction coordinator metadata schemas to use generated protocol Key: KAFKA-10497 URL: https://issues.apache.org/jira/browse/KAFKA-10497

Re: [DISCUSS] KIP-663: API to Start and Shut Down Stream Threads and to Request Closing of Kafka Streams Clients

Hi Sophie, Thank you for the feedback! I replied inline. Best, Bruno On 16.09.20 19:19, Sophie Blee-Goldman wrote: We guarantee that the metadata of the dead stream threads will be returned by KafkaStreams#localThreadsMetadata() at least until the next call to KafkaStreams#addStreamThread()

[jira] [Created] (KAFKA-10496) Create an in-memory DNS server for ClientUtilsTest and ClusterConnectionStatesTest

Justine Olshan created KAFKA-10496: -- Summary: Create an in-memory DNS server for ClientUtilsTest and ClusterConnectionStatesTest Key: KAFKA-10496 URL: https://issues.apache.org/jira/browse/KAFKA-10496

Re: KIP-669: Preserve Source Partition in Kafka Streams from context

Thanks Matthias for the reply I think i like the idea of the ability to use Record context in the default partitioner itself. I will join the discussion for KIP 478 to understand the context. On 2020/09/11 21:31:46, "Matthias J. Sax" wrote: > With regard to KIP-478, there is the idea to introd

Re: KIP-669: Preserve Source Partition in Kafka Streams from context

Thanks for taking time to reply. I thought about the confusion with overloads. creating another Partitioner seemed liked a good idea to start with, soon i realized the partitioner is interface written in a way which does not support anything other than key/value. It seems to me the idea of using

[jira] [Created] (KAFKA-10495) Fix spelling mistake

欧阳能达 created KAFKA-10495: Summary: Fix spelling mistake Key: KAFKA-10495 URL: https://issues.apache.org/jira/browse/KAFKA-10495 Project: Kafka Issue Type: Improvement Components: clients, c

[jira] [Created] (KAFKA-10494) Streams: enableSendingOldValues should not call parent if node is itself materialized

Andy Coates created KAFKA-10494: --- Summary: Streams: enableSendingOldValues should not call parent if node is itself materialized Key: KAFKA-10494 URL: https://issues.apache.org/jira/browse/KAFKA-10494 P