[DISCUSS] KIP-1112: allow custom processor wrapping

Hey all, We have a short KIP we'd like to propose which will allow injecting custom code modules around the processors of Kafka Streams applications, including DSL-built topologies. Please let us know if you have any thoughts or concerns https://cwiki.apache.org/confluence/display/KAFKA/KIP-111

[jira] [Created] (KAFKA-18026) Allow custom processor wrapping

A. Sophie Blee-Goldman created KAFKA-18026: -- Summary: Allow custom processor wrapping Key: KAFKA-18026 URL: https://issues.apache.org/jira/browse/KAFKA-18026 Project: Kafka Issue Typ

Re: [VOTE] KIP-1106: Add duration based offset reset option for consumer clients

+1 (binding) On 11/15/24 11:50 AM, Jun Rao wrote: Hi, ManiKumar, Thanks for the KIP. +1 Should we document that duration can't be negative in the following too? AutoOffsetReset duration(Duration duration) Jun On Thu, Nov 14, 2024 at 5:31 AM Manikumar wrote: Hi All, I would like to start

Re: [DISCUSS] KIP-1071: Streams Rebalance Protocol

Thanks for the updates! Few minor questions before we wrap this up and move to a vote: S1. Can you clarify the outward-facing behavior of a group that enters the NOT_READY state due to, say, missing source topics? It sounds like the application will continue to run without processing anything, whi

Re: [DISCUSS] KIP-1109: Unifying Kafka Consumer Topic Metrics

Moreover on a second thought, as this should be a correction in emitted topic names then we should not require a KIP per say but just fix it as part of jira itself. Please let me know your thoughts. Regards, Apoorv Mittal On Fri, Nov 15, 2024 at 10:16 PM Apoorv Mittal wrote: > Hi All, > I hav

Re: [DISCUSS] KIP-1109: Unifying Kafka Consumer Topic Metrics

Hi All, I have added the Context section in the KIP as per the discussion on the jira . Please let me know if there is any feedback or I shall directly start the voting thread. Regards, Apoorv Mittal On Tue, Nov 12, 2024 at 12:49 PM Apoorv Mitta

Re: [VOTE] KIP-1106: Add duration based offset reset option for consumer clients

Hi, ManiKumar, Thanks for the KIP. +1 Should we document that duration can't be negative in the following too? AutoOffsetReset duration(Duration duration) Jun On Thu, Nov 14, 2024 at 5:31 AM Manikumar wrote: > Hi All, > > I would like to start start vote on KIP-1106: > > https://cwiki.apache

[jira] [Resolved] (KAFKA-17948) Potential issue during tryComplete and onComplete simultaneous calls to access global variables

[ https://issues.apache.org/jira/browse/KAFKA-17948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao resolved KAFKA-17948. - Fix Version/s: 4.0.0 Resolution: Fixed merged the PR to trunk > Potential issue during tryComple

[jira] [Created] (KAFKA-18025) Flaky ShareConsumerTest testAcquisitionLockTimeoutOnConsumer

Lianet Magrans created KAFKA-18025: -- Summary: Flaky ShareConsumerTest testAcquisitionLockTimeoutOnConsumer Key: KAFKA-18025 URL: https://issues.apache.org/jira/browse/KAFKA-18025 Project: Kafka

[jira] [Reopened] (KAFKA-17809) Fix flaky test: testExplicitAcknowledgementCommitAsync

[ https://issues.apache.org/jira/browse/KAFKA-17809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apoorv Mittal reopened KAFKA-17809: --- > Fix flaky test: testExplicitAcknowledgementCommitAsync > -

Re: [VOTE] KIP-1106: Add duration based offset reset option for consumer clients

+1 (binding). Thanks for the KIP! On Fri, Nov 15, 2024 at 5:48 PM Lianet M. wrote: > Thanks for the KIP Manikumar! Nice improvement. > > +1 (binding) > > Lianet > > On Fri, Nov 15, 2024 at 11:46 AM David Arthur wrote: > > > Thanks! > > > > +1 binding > > > > On Fri, Nov 15, 2024 at 11:26 AM Man

[jira] [Resolved] (KAFKA-17791) Dockerfile should use `requirements.txt` to ensure dependencies

[ https://issues.apache.org/jira/browse/KAFKA-17791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mickael Maison resolved KAFKA-17791. Fix Version/s: 4.0.0 Resolution: Fixed > Dockerfile should use `requirements.txt` t

Re: [VOTE] KIP-1106: Add duration based offset reset option for consumer clients

Thanks for the KIP Manikumar! Nice improvement. +1 (binding) Lianet On Fri, Nov 15, 2024 at 11:46 AM David Arthur wrote: > Thanks! > > +1 binding > > On Fri, Nov 15, 2024 at 11:26 AM Manikumar > wrote: > > > Hi David. > > > > Thanks for the review. Yes, we will disallow the negative duration.

Re: [VOTE] KIP-1106: Add duration based offset reset option for consumer clients

Thanks! +1 binding On Fri, Nov 15, 2024 at 11:26 AM Manikumar wrote: > Hi David. > > Thanks for the review. Yes, we will disallow the negative duration. Updated > the KIP. > > > Thanks, > > On Fri, Nov 15, 2024 at 9:33 PM David Arthur wrote: > > > Mani, thanks for this KIP. It looks quite usef

Re: [VOTE] KIP-1106: Add duration based offset reset option for consumer clients

Hi David. Thanks for the review. Yes, we will disallow the negative duration. Updated the KIP. Thanks, On Fri, Nov 15, 2024 at 9:33 PM David Arthur wrote: > Mani, thanks for this KIP. It looks quite useful. > > I had one small question -- what happens if someone gives a negative > duration? F

Re: [VOTE] KIP-1106: Add duration based offset reset option for consumer clients

Mani, thanks for this KIP. It looks quite useful. I had one small question -- what happens if someone gives a negative duration? From what I can tell this is allowed in the ISO8601 spec. Should we explicitly disallow these, or perhaps auto-convert them to positive durations (via Duration#abs)? -D

Re: [DISCUSS] KIP-1099: Extend kafka-consumer-groups command line tool to support new consumer group

Hi, DJ2: Using "-" sounds good to me. DJ3: That seems reasonable to me. DJ4: Why not add it right now? I don't want to change the output of the tool too many times. Best, David On Fri, Nov 15, 2024 at 3:23 PM PoAn Yang wrote: > Hi David / Andrew, > > Thanks for review. Thanks Andrew for pick

[jira] [Created] (KAFKA-18024) ConcurrentModificationException in Kafka OffsetFetcher - Proposal for Thread-Safety Fix

Kamil Adam Nowak created KAFKA-18024: Summary: ConcurrentModificationException in Kafka OffsetFetcher - Proposal for Thread-Safety Fix Key: KAFKA-18024 URL: https://issues.apache.org/jira/browse/KAFKA-18024

Re: [DISCUSS] KIP-1099: Extend kafka-consumer-groups command line tool to support new consumer group

Hi David / Andrew, Thanks for review. Thanks Andrew for picking up kafka-share-groups.sh implementation. I will handle kafka-consumer-groups.sh. DJ3: After discussing with @Chia-Ping Tsai, we think that using new format is more clear. The new format will be like :,;:, Using colon(:) to conca

Re: [VOTE] KIP-1106: Add duration based offset reset option for consumer clients

Hi Manikumar, Thanks for the KIP! Duration based offset reset is a useful addition when remote storage is enabled. +1 (binding) Thanks, Kamal On Thu, Nov 14, 2024 at 7:51 PM Apoorv Mittal wrote: > Hi Manikumar, > Thanks for the KIP. It will be a valuable addition. > > +1 (non-binding) > > Reg

[jira] [Resolved] (KAFKA-18006) add 3.9.0 to end-to-end test

[ https://issues.apache.org/jira/browse/KAFKA-18006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruno Cadonna resolved KAFKA-18006. --- Resolution: Fixed > add 3.9.0 to end-to-end test > > >

[jira] [Resolved] (KAFKA-7720) kafka-configs script should also describe default broker entries

[ https://issues.apache.org/jira/browse/KAFKA-7720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Edoardo Comar resolved KAFKA-7720. -- Resolution: Implemented > kafka-configs script should also describe default broker entries > --

Re: [DISCUSS] KIP-1098: Reverse Checkpointing in MirrorMaker

Hi Vidor, KV2 and KV3. I think it is a common setup, but it is usually described as separating the produce and the consume workload. Instead of "prod" and "backup" think of "produce cluster" and "consume cluster". Currently, even if a group does not exist in the target cluster, both checkpointing

Re: How to push forward KIP-1011 complete implementation

Hi Ziming, I just read the KIP again and I don't see any deprecation period mentioned in the KIP. Where is it? I only see that we wanted to have a failback prior to 4.0 and we wanted to remove it in 4.0. However, I don't see anything mentioning that we would actually warn the user about it. Best,

Re: [DISCUSS] KIP-1098: Reverse Checkpointing in MirrorMaker

Hi Daniel, KV1. Thanks for the rundown, good to know that the impact is not concerning, I agree with the optimization not worth the effort KV2 and KV3. Here’s the setup I was thinking about. Suppose we have a topic on a prod cluster A that is replicated to a backup cluster B. There is a CG that

Re: [VOTE] KIP-1103: Additional metrics for cooperative consumption

Thanks everyone for the feedback and votes! I have added the voting thread on the KIP as mentioned by Chia, also renamed group_name to group_id as mentioned by Andrew. The KIP-1103 has passed with 4 binding votes (Chia, Jun, Manikumar and David A) and 1 non-binding vote (Andrew). Regards, Apoorv

Re: [DISCUSS] KIP-1098: Reverse Checkpointing in MirrorMaker

Hi Viktor, SVV3. In the current proposal, if the TopicFilter is not provided, it enables a different logic for reverse checkpointing - the task relies on the ReplicationPolicy to detect if a topic in the source cluster is a replica originating from the target cluster, e.g. in the A->B flow, the to

[jira] [Resolved] (KAFKA-17977) Remove new_consumer from E2E

[ https://issues.apache.org/jira/browse/KAFKA-17977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai resolved KAFKA-17977. Fix Version/s: 4.0.0 Resolution: Fixed > Remove new_consumer from E2E > ---

[DISCUSS] KIP-1111: Enforcing Explicit Naming for Kafka Streams Internal Topics

Hi Everyone, I would like to start a discussion on KIP-: Enforcing Explicit Naming for Kafka Streams Internal Topics This proposal aims to add a configuration that preve

[jira] [Created] (KAFKA-18023) Enforcing Explicit Naming for Kafka Streams Internal Topics (KIP-1111)

Sebastien Viale created KAFKA-18023: --- Summary: Enforcing Explicit Naming for Kafka Streams Internal Topics (KIP-) Key: KAFKA-18023 URL: https://issues.apache.org/jira/browse/KAFKA-18023 Project: