Re: [DISCUSS] KIP-1106: Add duration based offset reset option for consumer clients

Hi Matthias. Thanks for the review 1. looks like the majority of us are leaning towards BY_DURATION naming. I have updated the same in the KIP. 2. Thanks. Updated the KIP to remove private/internal implementation. Thanks, On Wed, Nov 13, 2024 at 6:28 AM Matthias J. Sax wrote: > Thanks for u

[jira] [Created] (KAFKA-18004) Use 3.8 to run zk service

Chia-Ping Tsai created KAFKA-18004: -- Summary: Use 3.8 to run zk service Key: KAFKA-18004 URL: https://issues.apache.org/jira/browse/KAFKA-18004 Project: Kafka Issue Type: Sub-task

[jira] [Created] (KAFKA-18003) add test to make sure `Admin#deleteRecords` can handle the corrupted records

Chia-Ping Tsai created KAFKA-18003: -- Summary: add test to make sure `Admin#deleteRecords` can handle the corrupted records Key: KAFKA-18003 URL: https://issues.apache.org/jira/browse/KAFKA-18003 Proj

[jira] [Resolved] (KAFKA-17922) add helper to ClusterInstance to create client component

[ https://issues.apache.org/jira/browse/KAFKA-17922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai resolved KAFKA-17922. Fix Version/s: 4.0.0 Resolution: Fixed > add helper to ClusterInstance to create cl

[jira] [Resolved] (KAFKA-16589) Consider removing `ClusterInstance#createAdminClient` since callers are not sure whether they need to call close

[ https://issues.apache.org/jira/browse/KAFKA-16589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai resolved KAFKA-16589. Resolution: Won't Fix KAFKA-17922 refactor all helpers so we don't need to address this no

[jira] [Resolved] (KAFKA-17978) StreamsUpgradeTest#test_rolling_upgrade_with_2_bounces system tests fail

[ https://issues.apache.org/jira/browse/KAFKA-17978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] PoAn Yang resolved KAFKA-17978. --- Resolution: Fixed > StreamsUpgradeTest#test_rolling_upgrade_with_2_bounces system tests fail > -

Re: [DISCUSS] KIP-1104: Allow Foreign Key Extraction from Both Key and Value in KTable Joins

I can just second was Lucas and Bill said already. 1. We cannot break compatibility 2. BiFunction sounds like a good alternative 3. I would personally deprecate the existing method, but don't feel strong about it. -Matthias On 11/12/24 8:33 AM, Bill Bejeck wrote: Hi Peter, It's important

[jira] [Reopened] (KAFKA-16949) System test test_dynamic_logging in connect_distributed_test is failing

[ https://issues.apache.org/jira/browse/KAFKA-16949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kirk True reopened KAFKA-16949: --- > System test test_dynamic_logging in connect_distributed_test is failing >

Re: [DISCUSS] KIP-1106: Add duration based offset reset option for consumer clients

Thanks for updating the KIP. I am happy to see that we seem to align to use a single config only :) Obviously, I need to bikeshed on the format: `BACK:` does not read well IMHO, and I think `auto.offset.reset="BACK_BY:"` would read much better. Andrew's suggestion of `BY_DURATION:` might ev

[jira] [Resolved] (KAFKA-17681) Fix unstable consumer_test.py#test_fencing_static_consumer

[ https://issues.apache.org/jira/browse/KAFKA-17681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kirk True resolved KAFKA-17681. --- Resolution: Cannot Reproduce > Fix unstable consumer_test.py#test_fencing_static_consumer >

[jira] [Resolved] (KAFKA-17962) test_pause_and_resume_sink fails with "Failed to consume messages after resuming sink connector" with CONSUMER group protocol

[ https://issues.apache.org/jira/browse/KAFKA-17962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kirk True resolved KAFKA-17962. --- Resolution: Cannot Reproduce > test_pause_and_resume_sink fails with "Failed to consume messages aft

[jira] [Created] (KAFKA-18002) Upgrade connect_distributed_test.py's test_exactly_once_source to support different group.protocol values

Kirk True created KAFKA-18002: - Summary: Upgrade connect_distributed_test.py's test_exactly_once_source to support different group.protocol values Key: KAFKA-18002 URL: https://issues.apache.org/jira/browse/KAFKA-1800

[jira] [Created] (KAFKA-18001) KafkaNetworkChannel missing UpdateRaftVoterRequestData logic

Alyssa Huang created KAFKA-18001: Summary: KafkaNetworkChannel missing UpdateRaftVoterRequestData logic Key: KAFKA-18001 URL: https://issues.apache.org/jira/browse/KAFKA-18001 Project: Kafka

[jira] [Created] (KAFKA-18000) Fix flaky ReplicaManager#testSuccessfulBuildRemoteLogAuxStateMetrics

David Arthur created KAFKA-18000: Summary: Fix flaky ReplicaManager#testSuccessfulBuildRemoteLogAuxStateMetrics Key: KAFKA-18000 URL: https://issues.apache.org/jira/browse/KAFKA-18000 Project: Kafka

[jira] [Created] (KAFKA-17999) Fix flaky DynamicConnectionQuotaTest testDynamicConnectionQuota

David Arthur created KAFKA-17999: Summary: Fix flaky DynamicConnectionQuotaTest testDynamicConnectionQuota Key: KAFKA-17999 URL: https://issues.apache.org/jira/browse/KAFKA-17999 Project: Kafka

[jira] [Created] (KAFKA-17998) Fix flaky OffloadAndTxnConsumeFromLeaderTest

David Arthur created KAFKA-17998: Summary: Fix flaky OffloadAndTxnConsumeFromLeaderTest Key: KAFKA-17998 URL: https://issues.apache.org/jira/browse/KAFKA-17998 Project: Kafka Issue Type: Test

Re: [DISCUSS] KIP-1050: Consistent error handling for Transactions

Thanks Lianet for review LM1 & LM2: We will extend parent classes only to maintain the hierarchy For TopicAuthorizationException and GroupAuthorizationException, we will extend parent class AuthorizationException. So new hierarchy will be "AuthorizationException < InvalidConfigurationException <

Re: [VOTE] KIP-1091: Improved Kafka Streams operator metrics

All, Quick update on KIP-1091. In an offline discussion, it was brought up that the existing JMX metric for the client state is named "state", so we'll update the name for the JMX thread state metric to "state" as well. Thanks, Bill On Mon, Nov 11, 2024 at 12:59 PM Bill Bejeck wrote: > Hi All

[jira] [Created] (KAFKA-17997) Remove deprecated config log.message.timestamp.difference.max.ms

Divij Vaidya created KAFKA-17997: Summary: Remove deprecated config log.message.timestamp.difference.max.ms Key: KAFKA-17997 URL: https://issues.apache.org/jira/browse/KAFKA-17997 Project: Kafka

Re: [DISCUSS] KIP-1104: Allow Foreign Key Extraction from Both Key and Value in KTable Joins

Hi Peter, It's important that we don't break compatibility. We faced a similar situation in KIP-149 when we provided access to the key in mapping and joining. I

[jira] [Resolved] (KAFKA-17744) Improve the State Updater logs when restoring state

[ https://issues.apache.org/jira/browse/KAFKA-17744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruno Cadonna resolved KAFKA-17744. --- Resolution: Fixed > Improve the State Updater logs when restoring state > --

[jira] [Created] (KAFKA-17996) kafka-metadata-quorum.sh add-controller cause the new added controller to crash with java.lang.IllegalArgumentException

Omnia Ibrahim created KAFKA-17996: - Summary: kafka-metadata-quorum.sh add-controller cause the new added controller to crash with java.lang.IllegalArgumentException Key: KAFKA-17996 URL: https://issues.apache.org/

Re: [DISCUSS] Require KIPs to include "How to teach this section"

Oh sorry for the noise all, I missed a couple subsequent responses that already addressed this. Den tis 12 nov. 2024 kl 16:05 skrev Anton Agestam : > Hi Matthias, > > Thanks for your input and for pointing that out. > > It at least is missing from this wiki page, and so is a link to the KIP > tem

Re: [DISCUSS] Require KIPs to include "How to teach this section"

Hi Matthias, Thanks for your input and for pointing that out. It at least is missing from this wiki page, and so is a link to the KIP template: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=50859233#KafkaImprovementProposals-WhatshouldbeincludedinaKIP ? Perhaps there should be

[jira] [Created] (KAFKA-17995) Large value for `retention.ms` could prevent remote data cleanup in Tiered Storage

Divij Vaidya created KAFKA-17995: Summary: Large value for `retention.ms` could prevent remote data cleanup in Tiered Storage Key: KAFKA-17995 URL: https://issues.apache.org/jira/browse/KAFKA-17995 Pr

Re: [DISCUSS] KIP-1106: Add duration based offset reset option for consumer clients

Thanks Manikumar for explaining, sounds good to me. Regards, Apoorv Mittal On Tue, Nov 12, 2024 at 1:47 PM Andrew Schofield < andrew_schofield_j...@outlook.com> wrote: > Hi, > Looks good now. Just one suggestion. > > AS8: Instead of "back:30D", I wonder whether the word 'duration' ought to > be

Re: [DISCUSS] KIP-1106: Add duration based offset reset option for consumer clients

Hi, Looks good now. Just one suggestion. AS8: Instead of "back:30D", I wonder whether the word 'duration' ought to be used to be consistent with kafka-consumer-groups.sh. So, "by-duration:P3D" or "duration:P3D" might be better. The overall idea of merging the configs into one config is fine in th

[jira] [Created] (KAFKA-17994) Runtime exceptions are not handled when deserializing kafka stream record

Ilya created KAFKA-17994: Summary: Runtime exceptions are not handled when deserializing kafka stream record Key: KAFKA-17994 URL: https://issues.apache.org/jira/browse/KAFKA-17994 Project: Kafka Is

[jira] [Created] (KAFKA-17993) reassign partition tool stuck with uncaught exception: 'value' field is too long to be serialized

Edoardo Comar created KAFKA-17993: - Summary: reassign partition tool stuck with uncaught exception: 'value' field is too long to be serialized Key: KAFKA-17993 URL: https://issues.apache.org/jira/browse/KAFKA-1799

Re: [DISCUSS] KIP-1106: Add duration based offset reset option for consumer clients

Hi Apoorv, AM7: AutoOffsetReset.java is for Kafka Streams API. I am not proposing any public Interface/class for Kafka Consumer. As mentioned in the KIP, even though OffsetResetStrategy is a public class, it's not used in any public APIs. I think new internal classes should be sufficient. AM8: Fi

Re: [DISCUSS] KIP-1104: Allow Foreign Key Extraction from Both Key and Value in KTable Joins

Hi, 1. I don't think we can/should break backwards compatibility. 2. Have you considered using `BiFunction foreignKeyExtractor` ? This should work without renaming the method. 3. I don't see the benefit of deprecating it. I agree, we wouldn't add both overloads normally, but the value-only overloa

[DISCUSS] KIP-1109: Unifying Kafka Consumer Topic Metrics

Hi All, I would like to start a discussion on KIP-1109: https://cwiki.apache.org/confluence/display/KAFKA/KIP-1109%3A+Unifying+Kafka+Consumer+Topic+Metrics This KIP streamlines topic and topic-partition metrics for Kafka Consumer, emitting the user defined topic name (as like kafka-producer). Reg

Re: [DISCUSS] KIP-1106: Add duration based offset reset option for consumer clients

Hi, I read the changes for single configuration and deprecated OffsetResetStrategy.java. AM7: Question: The KIP says that previous supported values were earliest/latest/none and new back: config would be added. We have no definition of "none" in the newly introduced AutoOffsetReset.java class henc

Re: [DISCUSS] KIP-1106: Add duration based offset reset option for consumer clients

Thanks Ismael and Lianet for the reviews. Based on suggestions, I have updated the KIP to in favour of having a single config (auto.offset.reset). I have also adopted the Lianet's suggestion on naming. auto.offset.reset=back:P3D -> reset back 3 days Let me know if there are any concerns. Than

Re: [ANNOUNCE] Apache Kafka 3.9.0

hi Josep > I guess you mean 3.9.0 right? Yes, sorry for my fat-fingering :( Josep Prat 於 2024年11月12日 週二 下午7:17寫道： > Hi Chia-Ping, > > I guess you mean 3.9.0 right? > > On Tue, Nov 12, 2024 at 12:16 PM Chia-Ping Tsai > wrote: > > > hi Colin > > > > 3.9.1 is nonexistent in the > > https://s3-us

Re: [ANNOUNCE] Apache Kafka 3.9.0

For a second I thought I missed a complete version :D On Tue, Nov 12, 2024 at 12:20 PM Chia-Ping Tsai wrote: > hi Josep > > > I guess you mean 3.9.0 right? > > Yes, sorry for my fat-fingering :( > > Josep Prat 於 2024年11月12日 週二 下午7:17寫道： > > > Hi Chia-Ping, > > > > I guess you mean 3.9.0 right?

Re: [ANNOUNCE] Apache Kafka 3.9.0

Hi Chia-Ping, I guess you mean 3.9.0 right? On Tue, Nov 12, 2024 at 12:16 PM Chia-Ping Tsai wrote: > hi Colin > > 3.9.1 is nonexistent in the > https://s3-us-west-2.amazonaws.com/kafka-packages > > Could you please check this? I'd like to add version 3.9.1 to the E2E > tests. > > Best, > Chia-P

Re: [ANNOUNCE] Apache Kafka 3.9.0

hi Colin 3.9.1 is nonexistent in the https://s3-us-west-2.amazonaws.com/kafka-packages Could you please check this? I'd like to add version 3.9.1 to the E2E tests. Best, Chia-Ping On 2024/11/07 23:11:51 Colin McCabe wrote: > The Apache Kafka community is pleased to announce the release for Apa

[jira] [Resolved] (KAFKA-17314) Fix the typo: `maxlifeTimeMs`

[ https://issues.apache.org/jira/browse/KAFKA-17314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai resolved KAFKA-17314. Fix Version/s: 4.0.0 Resolution: Fixed > Fix the typo: `maxlifeTimeMs` > --

[jira] [Created] (KAFKA-17992) Remove `getUnderlying` and `isKRaftTest` from ClusterInstance

Chia-Ping Tsai created KAFKA-17992: -- Summary: Remove `getUnderlying` and `isKRaftTest` from ClusterInstance Key: KAFKA-17992 URL: https://issues.apache.org/jira/browse/KAFKA-17992 Project: Kafka

[jira] [Created] (KAFKA-17991) Timed calls to future.get in DefaultStatePersister and test improvements

Sushant Mahajan created KAFKA-17991: --- Summary: Timed calls to future.get in DefaultStatePersister and test improvements Key: KAFKA-17991 URL: https://issues.apache.org/jira/browse/KAFKA-17991 Projec