Re: [DISCUSS] Apache Kafka 3.8.0 release

Hi all, We are now past the code freeze for the 3.8.0 release. If you think a commit should be backported to the 3.8 branch, please ping me in the PR (@jlprat). Thanks! On Wed, Jun 12, 2024 at 7:22 PM José Armando García Sancio wrote: > Hi Josep, > > See my comment below. > > On Wed, Jun 12, 2

[jira] [Resolved] (KAFKA-16901) add unit tests for ConsumerRecords#records(String)

[ https://issues.apache.org/jira/browse/KAFKA-16901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai resolved KAFKA-16901. Fix Version/s: 3.9.0 Resolution: Fixed > add unit tests for ConsumerRecords#records

[jira] [Created] (KAFKA-16948) Reset tier lag metrics on becoming follower

Kamal Chandraprakash created KAFKA-16948: Summary: Reset tier lag metrics on becoming follower Key: KAFKA-16948 URL: https://issues.apache.org/jira/browse/KAFKA-16948 Project: Kafka I

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 3.8 #37

See

Jenkins build is unstable: Kafka » Kafka Branch Builder » trunk #3002

See

[jira] [Resolved] (KAFKA-16938) non-dynamic props gets corrupted due to circular reference between DynamicBrokerConfig and DynamicConfig

[ https://issues.apache.org/jira/browse/KAFKA-16938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai resolved KAFKA-16938. Resolution: Fixed > non-dynamic props gets corrupted due to circular reference between >

Re: [DISCUSS] KIP-655: Add deduplication processor in kafka-streams

106: For the use-case of deduplicating a "at least once written" stream, we are sure that the duplicate record has the same key as the original one, and will land on the same task. Here, a user would want to specify a deduplication key different from the topic's key in case the topic's key is no

Re: Subscribe for discussion on Kafka® development

This is self service. Please follow the instructions from the web page: https://kafka.apache.org/contact -Matthias On 6/11/24 9:38 AM, Punsak Incham wrote: I want to subscribe for discussion on Kafka® development. Best regards, A black and blue logo Description automatically generated

Re: [VOTE] KIP-1035: StateStore managed changelog offsets

Thanks Nick. +1 (binding) Looking forward to get this all merged! -Matthias On 6/12/24 9:36 AM, Nick Telford wrote: Hi everyone, I'd like to call a vote on KIP-1035[1]. Regards, Nick 1: https://cwiki.apache.org/confluence/display/KAFKA/KIP-1035%3A+StateStore+managed+changelog+offsets

Re: [VOTE] KIP-1049: Add config log.summary.interval.ms to Kafka Streams

+1 (binding) On 6/11/24 1:17 PM, Sophie Blee-Goldman wrote: +1 (binding) Thanks for the KIP! On Tue, Jun 11, 2024 at 5:37 AM jiang dou wrote: HI I would like to start a vote for KIP-1049: Add config log.summary.interval.ms to Kafka Streams KIP： https://cwiki.apache.org/confluence/display/

Re: [DISCUSS] KIP-1027 Add MockFixedKeyProcessorContext

I believe the class name was picked on purpose, to make clear that it should not be used -- the problem is, that the class is in a public package and is by itself public (that's unfortunately require, given how Java works). Of course, it's also in the JavaDocs that the class is internal and s

Re: [DISCUSS] KIP-1056: Remove `default.` prefix for exception handler StreamsConfig

Thanks Greg, this is a valid idea. However, there was no demand in the past to allow for error handler callbacks in a fine grained manner, and I am frankly also not sure if it would make sense or would be required. Thus, I am not concerned about this case, and believe this KIP does make sens

[jira] [Resolved] (KAFKA-14877) refactor InMemoryLeaderEpochCheckpoint

[ https://issues.apache.org/jira/browse/KAFKA-14877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Satish Duggana resolved KAFKA-14877. Resolution: Invalid > refactor InMemoryLeaderEpochCheckpoint > ---

[jira] [Reopened] (KAFKA-16935) Automatically wait for cluster startup in embedded Connect integration tests

[ https://issues.apache.org/jira/browse/KAFKA-16935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Egerton reopened KAFKA-16935: --- > Automatically wait for cluster startup in embedded Connect integration tests > ---

[jira] [Resolved] (KAFKA-16935) Automatically wait for cluster startup in embedded Connect integration tests

[ https://issues.apache.org/jira/browse/KAFKA-16935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Egerton resolved KAFKA-16935. --- Resolution: Fixed > Automatically wait for cluster startup in embedded Connect integration t

[jira] [Created] (KAFKA-16947) Kafka Tiered Storage V2

Satish Duggana created KAFKA-16947: -- Summary: Kafka Tiered Storage V2 Key: KAFKA-16947 URL: https://issues.apache.org/jira/browse/KAFKA-16947 Project: Kafka Issue Type: Improvement Affec

[jira] [Resolved] (KAFKA-13561) Consider deprecating `StreamsBuilder#build(props)` function

[ https://issues.apache.org/jira/browse/KAFKA-13561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax resolved KAFKA-13561. - Resolution: Duplicate > Consider deprecating `StreamsBuilder#build(props)` function > --

[jira] [Resolved] (KAFKA-16890) Failing to build aux state on broker failover

[ https://issues.apache.org/jira/browse/KAFKA-16890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Satish Duggana resolved KAFKA-16890. Resolution: Fixed > Failing to build aux state on broker failover > --

[jira] [Created] (KAFKA-16946) Utils.getHost/getPort cannot parse SASL_PLAINTEXT://host:port

Luke Chen created KAFKA-16946: - Summary: Utils.getHost/getPort cannot parse SASL_PLAINTEXT://host:port Key: KAFKA-16946 URL: https://issues.apache.org/jira/browse/KAFKA-16946 Project: Kafka Issu

Kafka certification

Hi! I am a Back End developer and I am looking for official kafka certifications, but for now I am just finding courses. If exists any official certification for Kafka developer, I would like to know. Best regards! Matheus P. S.

Re: [DISCUSS] KIP-1027 Add MockFixedKeyProcessorContext

Hi Matthias, I think that strategy definitely works, abstracting away changes to FixedKeyRecord from users, I can put that new factory class and update the KIP. This might be a discussion for another KIP, but would it also make sense to rename the InternalFixedKeyRecordFactory to just FixedKeyRe

[jira] [Created] (KAFKA-16945) Cleanup StreamsBuilder and TopologyConfig

Matthias J. Sax created KAFKA-16945: --- Summary: Cleanup StreamsBuilder and TopologyConfig Key: KAFKA-16945 URL: https://issues.apache.org/jira/browse/KAFKA-16945 Project: Kafka Issue Type: I

[jira] [Resolved] (KAFKA-16897) Move OffsetIndexTest and OffsetMapTest to storage module

[ https://issues.apache.org/jira/browse/KAFKA-16897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai resolved KAFKA-16897. Fix Version/s: 3.9.0 Resolution: Fixed > Move OffsetIndexTest and OffsetMapTest to

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 3.8 #36

See

Re: [PR] MINOR: Mention confirming email subscriptions to contact page [kafka-site]

gharris1727 merged PR #606: URL: https://github.com/apache/kafka-site/pull/606 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kafka.apa

Re: Subscribe for discussion on Kafka(R) development

Hi Punsak, Please find the instructions for subscribing to this mailing list here: https://kafka.apache.org/contact and here: https://www.apache.org/foundation/mailinglists Thanks, Greg On Wed, Jun 12, 2024 at 7:11 AM Punsak Incham wrote: > I want to subscribe for discussion on Kafka® developm

[PR] MINOR: Mention confirming email subscriptions to contact page [kafka-site]

gharris1727 opened a new pull request, #606: URL: https://github.com/apache/kafka-site/pull/606 Some users partially subscribe to the mailing list, possibly because we don't mention the second step of replying to the confirmation email. Mentioning this succinctly in the page and linking to

Re: [DISCUSS] KIP-1056: Remove `default.` prefix for exception handler StreamsConfig

Hi Murali, Thanks for the KIP! I'm not familiar with Streams so I'll pose a general question, open for anyone to answer: The configs that are being changed don't currently accept in-place overwrites in the code, so the "default.*" prefix doesn't make sense. Could there be a KIP to accept in-plac

[jira] [Resolved] (KAFKA-16858) Flatten SMT throws NPE

[ https://issues.apache.org/jira/browse/KAFKA-16858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Greg Harris resolved KAFKA-16858. - Resolution: Fixed > Flatten SMT throws NPE > -- > > Key: KAF

[jira] [Created] (KAFKA-16944) Range assignor doesn't co-partition with stickiness

Ritika Reddy created KAFKA-16944: Summary: Range assignor doesn't co-partition with stickiness Key: KAFKA-16944 URL: https://issues.apache.org/jira/browse/KAFKA-16944 Project: Kafka Issue Typ

RE: [VOTE] KIP-1040: Improve handling of nullable values in InsertField, ExtractField, and other transformations

Hi all, I have a few questions. If you include all the related SMTs in the KIP, such as MaskField and ValueToKey, As a user, when will I be able to use the updated SMTs? Do all the related SMTs in the KIP need to be fixed first before you release them for users like me to use? Punsak Incham

[jira] [Created] (KAFKA-16943) Synchronously verify Connect worker startup failure in InternalTopicsIntegrationTest

Chris Egerton created KAFKA-16943: - Summary: Synchronously verify Connect worker startup failure in InternalTopicsIntegrationTest Key: KAFKA-16943 URL: https://issues.apache.org/jira/browse/KAFKA-16943

Re: [DISCUSS] Apache Kafka 3.8.0 release

Hi Josep, See my comment below. On Wed, Jun 12, 2024 at 1:17 PM Josep Prat wrote: > How long do you think it will take to bring KIP-853 to completion? We are still missing a few issues/jiras that need to get implemented for the feature to be usable. I would say a few more weeks. May be early Ju

Jenkins build is unstable: Kafka » Kafka Branch Builder » 3.8 #35

See

Re: [DISCUSS] Apache Kafka 3.8.0 release

Hi Colin, How long do you think it will take to bring KIP-853 to completion? Best, On Wed, Jun 12, 2024 at 6:42 PM Colin McCabe wrote: > Hi Josep, > > KIP-966 is not needed for feature parity with ZK. Indeed, KIP-966 will not > be implemented in ZK at all (and never was). > > There is a PR rel

Re: [VOTE] KIP-1040: Improve handling of nullable values in InsertField, ExtractField, and other transformations

Hi Mario, Thank you for mentioning the earlier DISCUSS thread. I found this comment from Chris, which was agreed upon and applied to the KIP: > Yes, I think we should just do one KIP for all the SMTs. You don't have to > implement everything all at once or by yourself, but I don't see why we > sh

Re: [DISCUSS] Apache Kafka 3.8.0 release

Hi Josep, KIP-966 is not needed for feature parity with ZK. Indeed, KIP-966 will not be implemented in ZK at all (and never was). There is a PR related to honoring dynamic configs for unclean.leader.election which we did agree to do for 3.8. It's under review and we hope to get it in soon. We

[VOTE] KIP-1035: StateStore managed changelog offsets

Hi everyone, I'd like to call a vote on KIP-1035[1]. Regards, Nick 1: https://cwiki.apache.org/confluence/display/KAFKA/KIP-1035%3A+StateStore+managed+changelog+offsets

Re: [DISCUSS] Apache Kafka 3.8.0 release

I'm just going to echo what Luke said. KIP-966 is definitely delayed to the next release and we only need the unclean leadership election PR he shared. Thanks, Justine On Wed, Jun 12, 2024 at 9:33 AM Calvin Liu wrote: > Hi Josep > Due to the limit of the review resources, KIP-966 may have to b

Re: [DISCUSS] Apache Kafka 3.8.0 release

Hi Josep Due to the limit of the review resources, KIP-966 may have to be delayed to the next release. Thanks. On Wed, Jun 12, 2024 at 2:05 AM Rajini Sivaram wrote: > > > > Yes, we could backport this one to the 3.8 branch. Would you be able to > do > > this once you merge this PR? > > > Thanks

Re: [DISCUSS] Apache Kafka 3.8.0 release

Hi Edoardo, Correct, you can still cherry-pick this one. I'll send an email tomorrow morning (CEST) asking maintainers to stop cherry picking commits unless we discuss it beforehand. Best, On Wed, Jun 12, 2024 at 6:09 PM Edoardo Comar wrote: > Hi Josep, I understand I am still in time to cher

Re: [VOTE] 3.7.1 RC1

Hi Igor, is it too late to cherry-pick https://github.com/apache/kafka/pull/16229 to 3.7.1 ? Thanks, Edo On Tue, 11 Jun 2024 at 08:29, Igor Soarez wrote: > > Now attaching the container build reports, which I seem to have forgotten to > include before. > > -- > Igor

Re: [DISCUSS] Apache Kafka 3.8.0 release

Hi Josep, I understand I am still in time to cherry-pick on 3.8.0 https://github.com/apache/kafka/pull/16229 right? thanks On Wed, 12 Jun 2024 at 11:34, Ivan Yurchenko wrote: > > Hi, > > I'll try to do all the fixes and changes for KIP-899 [1] sooner today, but > please proceed with the release

[jira] [Created] (KAFKA-16942) Use ConcurrentHashMap in RecordAccumulator#nodeStats

Kuan Po Tseng created KAFKA-16942: - Summary: Use ConcurrentHashMap in RecordAccumulator#nodeStats Key: KAFKA-16942 URL: https://issues.apache.org/jira/browse/KAFKA-16942 Project: Kafka Issue

[jira] [Resolved] (KAFKA-16865) Admin.describeTopics behavior change after KIP-966

[ https://issues.apache.org/jira/browse/KAFKA-16865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mickael Maison resolved KAFKA-16865. Resolution: Fixed > Admin.describeTopics behavior change after KIP-966 > -

Subscribe for discussion on Kafka® development

I want to subscribe for discussion on Kafka® development. Best regards, [A black and blue logo Description automatically generated] Punsak Incham (Mon) Data Platform Engineer Data Modernization Tel : +66(0)91-790-1302 | Email : pun...@mfec.co.th [signature_27890

Re: [VOTE] KIP-1040: Improve handling of nullable values in InsertField, ExtractField, and other transformations

Hi Greg, The same check was made by Chris Egerton during the discussion thread. The original KIP scope was just for the InsertField, ExtractField SMTs, then we decided to enlarge the scope of only the KIP to also other potential affected SMTs. As of now the PR scope, instead, is only for InsertF

[jira] [Created] (KAFKA-16941) Flaky test - testDynamicBrokerConfigUpdateUsingKraft [1] Type=Raft-Combined, MetadataVersion=4.0-IV0,Security=PLAINTEXT – kafka.admin.ConfigCommandIntegrationTest

Igor Soarez created KAFKA-16941: --- Summary: Flaky test - testDynamicBrokerConfigUpdateUsingKraft [1] Type=Raft-Combined, MetadataVersion=4.0-IV0,Security=PLAINTEXT – kafka.admin.ConfigCommandIntegrationTest Key: KAFKA-16941

Re: [DISCUSS] Apache Kafka 3.8.0 release

Hi, I'll try to do all the fixes and changes for KIP-899 [1] sooner today, but please proceed with the release if I don't manage. Ivan [1] https://github.com/apache/kafka/pull/13277 On Wed, Jun 12, 2024, at 12:54, Josep Prat wrote: > Hi Luke, > I think Jose, also mentioned that it won't be rea

[jira] [Created] (KAFKA-16940) Flaky test "testNoConsumeWithDescribeAclViaSubscribe(String).quorum=kraft" – kafka.api.DelegationTokenEndToEndAuthorizationWithOwnerTest

Igor Soarez created KAFKA-16940: --- Summary: Flaky test "testNoConsumeWithDescribeAclViaSubscribe(String).quorum=kraft" – kafka.api.DelegationTokenEndToEndAuthorizationWithOwnerTest Key: KAFKA-16940 URL: https://iss

Re: [DISCUSS] RemoteLogSegmentMetadataRecord Schema Evolution

Ignore the above message. Went over the Kafka message definition README guide and learnt about the tagged fields. On Tue, Jun 11, 2024 at 10:32 PM Kamal Chandraprakash < kamal.chandraprak...@gmail.com>

Re: [DISCUSS] Apache Kafka 3.8.0 release

Hi Luke, I think Jose, also mentioned that it won't be ready for v3.8.0 (but he can confirm this). My question now would be, given that it seems we would need a v3.9.0, do you think it's important to include https://github.com/apache/kafka/pull/16284 in v3.8.0? Best, On Wed, Jun 12, 2024 at 11:40

[jira] [Created] (KAFKA-16939) Revisit ConfigCommandIntegrationTest

Chia-Ping Tsai created KAFKA-16939: -- Summary: Revisit ConfigCommandIntegrationTest Key: KAFKA-16939 URL: https://issues.apache.org/jira/browse/KAFKA-16939 Project: Kafka Issue Type: Test

Re: [DISCUSS] Apache Kafka 3.8.0 release

Hi Josep For KIP-966, I think Calvin had mentioned he won't complete in v3.8.0. https://lists.apache.org/thread/fsnr8wy5fznzfso7jgk90skgyo277fmw For unclean leader election, all we need is this PR: https://github.com/apache/kafka/pull/16284 For this PR, I think it needs one more week to be comple

Jenkins build is unstable: Kafka » Kafka Branch Builder » trunk #3000

See

Re: [DISCUSS] Apache Kafka 3.8.0 release

> > Yes, we could backport this one to the 3.8 branch. Would you be able to do > this once you merge this PR? Thanks Josep! There were some unit test failures in last night's PR build, so I have commented on the PR for Ivan to fix. If he fixes it today, then I can merge to trunk and then cherry-p

Re: [DISCUSS] Apache Kafka 3.8.0 release

Hi all, We are now really close to the planned code freeze deadline (today EOD). According to KIP-1012 [1] we agreed to stay in the 3.x branch until we achieve feature parity regarding Zookeeper and KRaft. The two main KIPs identified that would achieve this are: KIP-853 [2] and KIP-966 [3]. At th

Re: [DISCUSS] Apache Kafka 3.8.0 release

Hi Rajini, Yes, we could backport this one to the 3.8 branch. Would you be able to do this once you merge this PR? Thanks On Tue, Jun 11, 2024 at 10:53 PM Rajini Sivaram wrote: > Hi Josep, > > The PR https://github.com/apache/kafka/pull/13277 for KIP-899 looks ready > to be merged (waiting for

[jira] [Created] (KAFKA-16938) non-dynamic props gets corrupted due to circular reference between DynamicBrokerConfig and DynamicConfig

Chia-Ping Tsai created KAFKA-16938: -- Summary: non-dynamic props gets corrupted due to circular reference between DynamicBrokerConfig and DynamicConfig Key: KAFKA-16938 URL: https://issues.apache.org/jira/browse/K