[jira] [Created] (KAFKA-17943) Error handling when calling replicaManager.getPartitionOrException

Abhinav Dixit created KAFKA-17943: - Summary: Error handling when calling replicaManager.getPartitionOrException Key: KAFKA-17943 URL: https://issues.apache.org/jira/browse/KAFKA-17943 Project: Kafka

Re: [DISCUSS] KIP-1101: Trigger rebalance on rack topology changes

Hi Chia-Ping, Thanks for the review and suggestions. Q0: Add how rack change and how it affects topic partition. Q1: Add why we need a balance algorithm to Motivation section. Q2: After checking again, we don’t need to update cache when we replay records. We only need to renew it in consumer h

Re: [VOTE] KIP-1102: Enable clients to rebootstrap based on timeout or error code

Hi Rajini, Thanks for the KIP. +1 (non-binding) Regards, Apoorv Mittal On Tue, Nov 5, 2024 at 10:46 AM Andrew Schofield < andrew_schofield_j...@outlook.com> wrote: > Hi Rajini, > Thanks for the KIP. > > +1 (non-binding) > > Andrew > > > From: Rajini Siv

Re: [DISCUSS] KIP-1103: Additional metrics for cooperative consumption

Thanks for the feedback Andrew. AS9: Yes, you are correct it's topic name and not topic id. AS10: Thanks, I have updated the description. Regards, Apoorv Mittal On Tue, Nov 5, 2024 at 10:56 AM Andrew Schofield < andrew_schofield_j...@outlook.com> wrote: > Hi Apoorv, > A couple of additional c

[jira] [Created] (KAFKA-17945) Use leader epoch while calling replicaManager.fetchOffsetForTimestamp in ShareFetchUtils

Chirag Wadhwa created KAFKA-17945: - Summary: Use leader epoch while calling replicaManager.fetchOffsetForTimestamp in ShareFetchUtils Key: KAFKA-17945 URL: https://issues.apache.org/jira/browse/KAFKA-17945

[jira] [Created] (KAFKA-17944) Add error handling in the cases when replicaManager.fetchOffsetForTimestamp throws an error

Chirag Wadhwa created KAFKA-17944: - Summary: Add error handling in the cases when replicaManager.fetchOffsetForTimestamp throws an error Key: KAFKA-17944 URL: https://issues.apache.org/jira/browse/KAFKA-17944

[DISCUSS] New label for breaking changes in 4.0

Hi folks We would be making many breaking changes in 4.0 such as removal of deprecated metrics, removal of configs etc. It would be nice if we can document all such backward incompatible behaviour in our documentation. The alternative is to leave the user with trying to figure out which changes a

[jira] [Resolved] (KAFKA-17819) Handle acknowledgements when subscription changes.

[ https://issues.apache.org/jira/browse/KAFKA-17819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shivsundar R resolved KAFKA-17819. -- Resolution: Fixed > Handle acknowledgements when subscription changes. > -

Re: [DISCUSS]KIP-1107: Adding record-level acks for producers

hi TaiJuWu, Q0: Could you please add getter (Short acks()) to "public interface" section? Q1: Could you please add RPC json reference to prove "been available at the RPC-level," Q2: Could you please add link to producer docs to prove "share a single producer instance across multiple threads"

[jira] [Resolved] (KAFKA-17877) IllegalStateException: missing producer id from the WriteTxnMarkersResponse

[ https://issues.apache.org/jira/browse/KAFKA-17877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Jacot resolved KAFKA-17877. - Fix Version/s: 4.0.0 Resolution: Fixed > IllegalStateException: missing producer id from

Re: [DISCUSS] KIP-1106: Add duration based offset reset option for consumer clients

Hi Manikumar, Thanks for the KIP. AS1: As Apoorv points out, the values of auto.offset.reset correspond to the enum OffsetResetStrategy. I would say that's a useful thing to maintain and I suggest adding a fourth value "by.duration" (BY_DURATION), and then another configuration for the duration

Re: [DISCUSS]KIP-1107: Adding record-level acks for producers

Sorry, please read: > LibRdKafka supports acks at the topic level, but then it does *NOT* group ProduceRequest for different topics in a single request.

[jira] [Resolved] (KAFKA-17248) Extend KIP-714 to allow for registering additional metrics with clients

[ https://issues.apache.org/jira/browse/KAFKA-17248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bill Bejeck resolved KAFKA-17248. - Resolution: Fixed > Extend KIP-714 to allow for registering additional metrics with clients > --

Re: [DISCUSS] New label for breaking changes in 4.0

I'd like to propose one more rule: 3. The PR should update `upgrade.html` ( https://github.com/apache/kafka/blob/trunk/docs/upgrade.html#L25) if it brings a breaking change. Best, Chia-Ping Divij Vaidya 於 2024年11月6日 週三 上午12:04寫道： > Hi folks > > We would be making many breaking changes in 4.0

Re: [DISCUSS] KIP-1106: Add duration based offset reset option for consumer clients

Hi All, Thanks for the inputs. I have updated the KIP based on the discussion so far. Few of the changes: 1. I have updated the KIP to use ISO-8601 format for defining duration. 2. Add support for the earliest-local option, which will reset to local-log-start-offset. 2. Changes to ConsumerGroupCo

[jira] [Resolved] (KAFKA-17609) Convert Kafka Streams system tests to use KRaft

[ https://issues.apache.org/jira/browse/KAFKA-17609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bill Bejeck resolved KAFKA-17609. - Resolution: Fixed > Convert Kafka Streams system tests to use KRaft > --

Re: [DISCUSS] KIP-1106: Add duration based offset reset option for consumer clients

Hi Dániel, Thanks for the review, DU1,DU2: Based on suggestions, I have updated the config option to support ISO format. DU3: We will compute the timestamp on the client side and use that timestamp to fetch the offset from the server. This will be similar to the kafka-consumer-group cmd's --by-

Re: [DISCUSS] KIP-1106: Add duration based offset reset option for consumer clients

Hi Apoorv, Thanks for the review. AM1, AM3: Yes, I agree. I have added OffsetResetStarategy class changes to KIP AM2: Yes, I agree. I have updated the KIP to use ISO format AM4: With ISO format, it's more flexible now. If required, we can easily add support for reset to a particular datetime( s

Re: [DISCUSS] KIP-1106: Add duration based offset reset option for consumer clients

Hi Manikumar, Thanks for the updates. A couple of other comments. AS4: I think the AK conventions would mean the config option should be "earliest_local" and "by_duration", and the enum constants should be "EARLIEST_LOCAL" and "BY_DURATION". AS5: The example P12M is invalid. PT12M (12 minutes) is

Re: [DISCUSS] KIP-1103: Additional metrics for cooperative consumption

Hi All, Please let me know if there is any other feedback or suggestions. Regards, Apoorv Mittal On Fri, Nov 1, 2024 at 5:30 PM Apoorv Mittal wrote: > Thanks Jun and Andrew for reviewing and feedback. > > J1: I was thinking of having metrics as per the APIs in > SharePartitionManager but I thi

[VOTE] KIP-1102: Enable clients to rebootstrap based on timeout or error code

Hi everyone, I would like to start vote on KIP-1102 ( https://cwiki.apache.org/confluence/display/KAFKA/KIP-1102%3A+Enable+clients+to+rebootstrap+based+on+timeout+or+error+code). This KIP extends KIP-899 by introducing a timeout configuration and error code that can be used to trigger rebootstrapp

[jira] [Created] (KAFKA-17942) Close producers and consumers in "finally" block in ShareConsumerTest.

Shivsundar R created KAFKA-17942: Summary: Close producers and consumers in "finally" block in ShareConsumerTest. Key: KAFKA-17942 URL: https://issues.apache.org/jira/browse/KAFKA-17942 Project: Kafka

Re: [DISCUSS]KIP-1107: Adding record-level acks for producers

Hi TaiJuWu, What would be the behavior if the application sends several records with different ack levels? The RPC support acks at the ProduceRequest, so common for several batches, might even be for different topic-partitions: https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka

Re: [DISCUSS] KIP-655: Add deduplication processor in kafka-streams

Hi Matthias, Thanks for your review, and sorry for this late reply. (102) Ok, I keep it open here. (104) > Well, but if we say punctuation records are "inherently distinct", > there is nothing to be de-duplicated? We still need to deduplicate punctuation records even if they are distinct. A f

Re: [VOTE] KIP-1102: Enable clients to rebootstrap based on timeout or error code

Hi Rajini, Thanks for the KIP. +1 (non-binding) Andrew From: Rajini Sivaram Sent: 05 November 2024 09:24 To: dev Subject: [VOTE] KIP-1102: Enable clients to rebootstrap based on timeout or error code Hi everyone, I would like to start vote on KIP-110

Re: [DISCUSS] KIP-1103: Additional metrics for cooperative consumption

Hi Apoorv, A couple of additional comments. AS9: The tag share-partition : {group-topic-partition} confirm the format. I suppose it's one tag `group-id:topic-name:partition`, and not topic ID. Topic ID is clearly more authoritative, but I don't think it's usable in metrics. AS10: The descriptio

[jira] [Resolved] (KAFKA-17870) Disallow creating too many partitions before using ReplicaPlacer

[ https://issues.apache.org/jira/browse/KAFKA-17870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin McCabe resolved KAFKA-17870. -- Fix Version/s: 4.0.0 Resolution: Fixed > Disallow creating too many partitions before u

Re: [DISCUSS] KIP-1106: Add duration based offset reset option for consumer clients

Hi Manikumar, Thanks for the changes. Just minor comment and a question: AM5: The description "How to initialize the share-partition start offset:" for "*share.auto.offset.reset*" seems incomplete as it ends with `:`. Should we write the details as defined in KIP-932 i.e. How to initialize the sh

Re: [DISCUSS] KIP-1106: Add duration based offset reset option for consumer clients

Hi Matthias, Thanks for the review. MS1: looks like KIP-842 is stuck in the voting. Current changes should fit into KIP-842 (few updates definitely required) as KIP-842 is proposing to add new configs (auto.offset.reset.on.no.initial.offset and auto.offset.reset.on.invalid.offset). I prefer to re

[jira] [Resolved] (KAFKA-17916) Convert Kafka Connect system tests to use KRaft

[ https://issues.apache.org/jira/browse/KAFKA-17916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kevin Wu resolved KAFKA-17916. -- Resolution: Fixed > Convert Kafka Connect system tests to use KRaft >

[jira] [Created] (KAFKA-17947) Update currentLag(), pause(), and resume() to perform logic in background thread

Kirk True created KAFKA-17947: - Summary: Update currentLag(), pause(), and resume() to perform logic in background thread Key: KAFKA-17947 URL: https://issues.apache.org/jira/browse/KAFKA-17947 Project: K

Apply to contribute Kafka

Hi, there Thanks for give me access to Kafka project, below is my information: Wiki ID: jerrycai JIRA ID: jerrycai Email: jerrycai...@gmail.com Github: https://github.com/JerryCai Thanks a lot! Jerry Cai

[jira] [Created] (KAFKA-17946) Flaky test DeafultStateUpdaterTest::shouldResumeStandbyTask due to concurrency issue

Ao Li created KAFKA-17946: - Summary: Flaky test DeafultStateUpdaterTest::shouldResumeStandbyTask due to concurrency issue Key: KAFKA-17946 URL: https://issues.apache.org/jira/browse/KAFKA-17946 Project: Kafka

[jira] [Resolved] (KAFKA-17941) TransactionStateManager.loadTransactionMetadata method may get stuck in an infinite loop

[ https://issues.apache.org/jira/browse/KAFKA-17941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao resolved KAFKA-17941. - Fix Version/s: 4.0.0 Resolution: Fixed merged the PR to trunk. > TransactionStateManager.loadTra

Re: [DISCUSS]KIP-1107: Adding record-level acks for producers

Hi Chia-Ping, Thanks for your feedback. I have updated KIP based on your suggestions. Best, Stanley Chia-Ping Tsai 於 2024年11月5日 週二 下午4:41寫道： > hi TaiJuWu, > > Q0: Could you please add getter (Short acks()) to "public interface" > section? > > Q1: Could you please add RPC json reference to prov

[VOTE] KIP-1091: Improved Kafka Streams operator metrics

Hi All, I'd like to call for a vote on KIP-1091 (discussion thread ) Thanks, Bill

Re: [DISCUSS]KIP-1107: Adding record-level acks for producers

Hi TaiJuWu, Thanks for the KIP! Can you explain in the KIP about the behavior when the number of acks is different for individual records? I think the current description using the word "straightforward" does little to explain that, and may actually be hiding some complexity. For example, the se

Jenkins build is still unstable: Kafka » Kafka PowerPC Daily » test-powerpc #109

See

Re: [DISCUSS]KIP-1107: Adding record-level acks for producers

Hi Fred and Greg, Apologies for the delayed response. Yes, you’re correct. I’ll outline the behavior I expect. Thanks for your feedback! Best, TaiJuWu Greg Harris 於 2024年11月6日 週三 上午9:48寫道： > Hi TaiJuWu, > > Thanks for the KIP! > > Can you explain in the KIP about the behavior when the numbe

[jira] [Resolved] (KAFKA-17931) Revise the `specifying-test-retries` section in README

[ https://issues.apache.org/jira/browse/KAFKA-17931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai resolved KAFKA-17931. Fix Version/s: 4.0.0 Resolution: Fixed > Revise the `specifying-test-retries` secti

Re: Apply to contribute Kafka

Hi Jerry, I've granted you wiki permissions. Let me know if you seem to be missing access to anything. Justine On Tue, Nov 5, 2024 at 3:38 PM Jerry Cai wrote: > Hi, there > > Thanks for give me access to Kafka project, below is my information: > > Wiki ID: jerrycai > JIRA ID: jerrycai > > Ema

[jira] [Resolved] (KAFKA-17936) Refactor SslTransportLayerTest#testDsaKeyPair to avoid skip test

[ https://issues.apache.org/jira/browse/KAFKA-17936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai resolved KAFKA-17936. Fix Version/s: 4.0.0 Resolution: Fixed > Refactor SslTransportLayerTest#testDsaKeyP

[jira] [Resolved] (KAFKA-17924) Remove `bufferpool-wait-time-total`, `io-waittime-total`, and `iotime-total`

[ https://issues.apache.org/jira/browse/KAFKA-17924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Divij Vaidya resolved KAFKA-17924. -- Resolution: Fixed > Remove `bufferpool-wait-time-total`, `io-waittime-total`, and `iotime-tota

Re: [VOTE] KIP-1058: Txn consumer exerts pressure on remote storage when reading non-txn topic

Had another read of the KIP, LGTM! +1 from me. Thanks for the KIP! Luke On Wed, Nov 6, 2024 at 1:58 PM Satish Duggana wrote: > +1 > > Thanks Kamal for the KIP. Discussed offline with Kamal on the proposal and > the changes. > It improves the performance of consumers' remote reads in read_commit

[jira] [Created] (KAFKA-17948) Potential problem during tryComplete and onComplete simultaneous calls

Abhinav Dixit created KAFKA-17948: - Summary: Potential problem during tryComplete and onComplete simultaneous calls Key: KAFKA-17948 URL: https://issues.apache.org/jira/browse/KAFKA-17948 Project: Kaf

Re: [DISCUSS] KIP-1105: Make remote log manager thread-pool configs dynamic

Hi Federico, Thanks for the review! 1. Changed the API name to `isReady` 2. Added an example of stacktrace in the KIP. PTAL. Thanks, Kamal On Mon, Nov 4, 2024 at 2:37 PM Federico Valeri wrote: > Hi Kamal, these changes make sense to me. Thanks. > > In this case, I wonder if "isReady" could b

Re: [DISCUSS] KIP-1106: Add duration based offset reset option for consumer clients

Hi Andrew, Thanks for the review. AS4, AS5: Updated the KIP. AS6: Yes, it applies to both. updated the KIP. Thanks, On Tue, Nov 5, 2024 at 10:58 PM Andrew Schofield wrote: > > Hi Manikumar, > Thanks for the updates. A couple of other comments. > > AS4: I think the AK conventions would mean

Re: [DISCUSS] KIP-1058: Txn consumer exerts pressure on remote storage when reading non-txn topic

Hi Divij, The implementation does not force the plugin to have an optimization. I've updated the public API javadoc about this change. PTAL. On Tue, Nov 5, 2024 at 12:42 AM Divij Vaidya wrote: > Sorry for the churn on this one. But circling back to the API. > > Today, the API forces the plugins

Re: [DISCUSS] KIP-1106: Add duration based offset reset option for consumer clients

Hi Apoorv, Thanks for the review. AM5: Thanks, Updated the KIP. AM6: Currently we can achieve this with the combination of offsetsForTimes() and seek() API.. Maybe we can add a generic API like seekToTimes(Map timestampsToSearch) if required. Let's see what others think. Thanks, On Wed, Nov 6

Re: [VOTE] KIP-1058: Txn consumer exerts pressure on remote storage when reading non-txn topic

+1 Thanks Kamal for the KIP. Discussed offline with Kamal on the proposal and the changes. It improves the performance of consumers' remote reads in read_committed mode. It brings minor changes in a backward compatible manner. ~Satish. On Mon, 4 Nov 2024 at 19:33, Divij Vaidya wrote: > +1 > >

[jira] [Resolved] (KAFKA-12827) Remove Deprecated method KafkaStreams#setUncaughtExceptionHandler

[ https://issues.apache.org/jira/browse/KAFKA-12827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax resolved KAFKA-12827. - Resolution: Fixed > Remove Deprecated method KafkaStreams#setUncaughtExceptionHandler >

Re: [DISCUSS]KIP-1107: Adding record-level acks for producers

Hi Fred and Greg, Thanks for your feedback and it really not straightforward but interesting! There are some behavior I expect. The current producer uses the *RecordAccumulator* to gather records, and the sender thread sends them in batches. We can track each record’s acknowledgment setting as it

Re: [DISCUSS] KIP-1105: Make remote log manager thread-pool configs dynamic

Thanks Kamal, LGTM, but you should replace all instances of isInitialized to isReady in the rest of the KIP. On Wed, Nov 6, 2024 at 5:22 AM Kamal Chandraprakash wrote: > > Hi Federico, > > Thanks for the review! > > 1. Changed the API name to `isReady` > 2. Added an example of stacktrace in the K