Re: [VOTE] KIP-1033: Add Kafka Streams exception handler for exceptions occurring during processing

Thanks Lianet, I’ll conclude the vote with 4 binding votes from Matthias J. Sax, Sophie Blee-Goldman, Bill Bejeck and Bruno Cadona. Thank you everyone! On Tue, 14 May 2024 at 17:56, Lianet M. wrote: > > +1 (non-binding) > > Thanks for the KIP and updates! > > Lianet > > On Tue, May 14, 2024, 12

Re: [DISCUSS] KIP-1038: Add Custom Error Handler to Producer

Hello Alieh, thanks for this useful KIP. There is a typo in the motivation when you talk about the UnknownTopicOrPartitionException. It's delivery.timeout.ms, not deliver.timeout.ms. In the past, I did some work to improve and clean the official Kafka examples, which I think are useful for new Ka

[jira] [Created] (KAFKA-16769) Delete deprecated add.source.alias.to.metrics configuration

Mickael Maison created KAFKA-16769: -- Summary: Delete deprecated add.source.alias.to.metrics configuration Key: KAFKA-16769 URL: https://issues.apache.org/jira/browse/KAFKA-16769 Project: Kafka

Re: [DISCUSS] Apache Kafka 3.8.0 release

Hi Kafka developers! Today is the KIP freeze deadline. All KIPs should be accepted by EOD today. Tomorrow morning (CEST timezone) I'll start summarizing all KIPs that have been approved. Please any KIP approved after tomorrow should be adopted in a future release version, not 3.8. Other relevant

Re: [DISCUSS] KIP-924: customizable task assignment for Streams

Hi, Thank you for the updates! I have a couple of comments. U1 Yeah, that makes sense to me. However, could we break out the assignment part to make it more readable? Something like: KafkaStreamsAssignment.for(processId).withAssignment(assignment).withFollowRebalance(rebalanceDeadline) nits

Re: [DISCUSS] KIP-950: Tiered Storage Disablement

Hi Christo, Thanks for the explanation. I think it would be good if you could add that into the KIP. Otherwise, LGTM. Thank you. Luke On Mon, May 13, 2024 at 11:55 PM Christo Lolov wrote: > Heya! > > re Kamal - Okay, I believe I understand what you mean and I agree. I have > made the followin

Re: [VOTE] KIP-950: Tiered Storage Disablement

Hi Christo, In addition to the minor comments left in the discussion thread, it LGTM. +1 from me. Thank you. Luke On Tue, May 14, 2024 at 11:21 PM Christo Lolov wrote: > Heya! > > I would like to start a vote on KIP-950: Tiered Storage Disablement in > order to catch the last Kafka release ta

Re: [VOTE] KIP-1036: Extend RecordDeserializationException exception

Hi, We can conclude with +4 binding and +3 non-binding votes. So it's adopted and I will propose it for the 3.8.0 release. Thank you everyone for your help and participation.

Re: [VOTE] KIP-950: Tiered Storage Disablement

Hi Christo, Thanks for the KIP and incorporating the review comments. Please update the KIP with the latest details. +1 (non-binding). Thanks, Kamal On Wed, May 15, 2024 at 3:18 PM Luke Chen wrote: > Hi Christo, > > In addition to the minor comments left in the discussion thread, it LGTM. >

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #2904

See

Re: [DISCUSS] Apache Kafka 3.8.0 release

Hey Josep! The KIP 1028 has received the required votes. Voting thread:- https://lists.apache.org/thread/cdq4wfv5v1gpqlxnf46ycwtcwk5wos4q But we are keeping the vote open for 72 hours as per the process. I would like to request you to please consider it for the 3.8.0 release. Thanks and regards,

[DISCUSS] KIP-1044: A proposal to change idempotent producer -- server implementation

This is a proposal that should solve the OOM problem on the servers without some of the other proposed KIPs being active. Full details in https://cwiki.apache.org/confluence/display/KAFKA/KIP-1044%3A+A+proposal+to+change+idempotent+producer+--+server+implementation

[jira] [Created] (KAFKA-16770) Coalesce records into bigger batches

David Jacot created KAFKA-16770: --- Summary: Coalesce records into bigger batches Key: KAFKA-16770 URL: https://issues.apache.org/jira/browse/KAFKA-16770 Project: Kafka Issue Type: Sub-task

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #2905

See

Re: [DISCUSS] KIP-1042 support for wildcard when creating new acls

Thank you Haruki for the feedback. There are certainly many such use cases where customers ended up creating custom authorizers to handle these MATCH based patterns. And regarding updating method matchingAcls, Claude updated the KIP mentioning an approach, which definitely optimizes the flow, but c

Re: [DISCUSS] KIP-891: Running multiple versions of Connector plugins

Hi Greg, Thanks for your responses! Continuations of existing discussions: Regarding crashing the worker on startup--yes, there is also a risk to allowing it to join the cluster. But this risk is already present with existing error cases, and I don't see anything that justifies changing existing

[jira] [Created] (KAFKA-16771) First log directory printed twice when formatting storage

Mickael Maison created KAFKA-16771: -- Summary: First log directory printed twice when formatting storage Key: KAFKA-16771 URL: https://issues.apache.org/jira/browse/KAFKA-16771 Project: Kafka

Re: [DISCUSS] Apache Kafka 3.7.1 release

Hi Igor, Thanks for volunteering, +1 Mickael On Thu, Apr 25, 2024 at 11:09 AM Igor Soarez wrote: > > Hi everyone, > > I'd like to volunteer to be the release manager for a 3.7.1 release. > > Please keep in mind, this would be my first release, so I might have some > questions, > and it might a

Re: [DISCUSS] KIP-1044: A proposal to change idempotent producer -- server implementation

Hi Claude Thanks for raising this KIP. It is an interesting idea. I had a quick review for the KIP and I have few notes 10. > The issue is that the number of PIDs that need to be tracked has exploded and > has resulted in OOM failures that cause the entire cluster to crash. There > are multip

Re: [DISCUSS] KIP-1044: A proposal to change idempotent producer -- server implementation

Also in the rejection alternatives you listed an approved KIP which is a bit confusing can you move this to motivations instead > On 15 May 2024, at 14:35, Claude Warren wrote: > > This is a proposal that should solve the OOM problem on the servers without > some of the other proposed KIPs bei

[jira] [Created] (KAFKA-16772) Introduce a type for kraft.version

José Armando García Sancio created KAFKA-16772: -- Summary: Introduce a type for kraft.version Key: KAFKA-16772 URL: https://issues.apache.org/jira/browse/KAFKA-16772 Project: Kafka

Re: [VOTE] KIP-1025: Optionally URL-encode clientID and clientSecret in authorization header

Hi, +1 (binding) Thanks for the KIP! Mickael On Sun, Apr 21, 2024 at 7:12 PM Nelson B. wrote: > > Hi all, > > Just a kind reminder. I would really appreciate if we could get two more > binding +1 votes. > > Thanks > > On Mon, Apr 8, 2024, 2:08 PM Manikumar wrote: > > > Thanks for the KIP. > >

Re: [DISCUSS] Apache Kafka 3.8.0 release

FWIW I think that the low blast radius for KIP-1028 should allow it to proceed without adhering to the usual KIP and feature freeze dates. Code freeze is probably worth still respecting, at least if changes are required to the docker/jvm/Dockerfile. But I defer to Josep's judgement as the release

Re: [DISCUSS] Apache Kafka 3.8.0 release

The KIP freeze is just about having the KIP accepted. Not sure why we would need an exception for that. Ismael On Wed, May 15, 2024 at 9:20 AM Chris Egerton wrote: > FWIW I think that the low blast radius for KIP-1028 should allow it to > proceed without adhering to the usual KIP and feature fr

Re: [DISCUSS] Apache Kafka 3.8.0 release

The KIP isn't officially accepted until its vote thread closes after being open for at least three days. KIP-1028 won't make that deadline. On Wed, May 15, 2024, 12:36 Ismael Juma wrote: > The KIP freeze is just about having the KIP accepted. Not sure why we would > need an exception for that. >

Re: [DISCUSS] Apache Kafka 3.8.0 release

Hi all, You still have the full day of today (independently for the timezone) to get KIPs approved. Tomorrow morning (CEST timezone) I'll send another email asking developers to assign future approved KIPs to another version that is not 3.8. So, the only problem I see with KIP-1028 is that it hasn

Re: [DISCUSS] Apache Kafka 3.8.0 release

Ah, I see Chris was faster writing this than me. On Wed, May 15, 2024 at 6:43 PM Josep Prat wrote: > Hi all, > You still have the full day of today (independently for the timezone) to > get KIPs approved. Tomorrow morning (CEST timezone) I'll send another email > asking developers to assign futu

Re: [DISCUSS] Apache Kafka 3.8.0 release

And my maths are wrong! I added 24 hours more to all the numbers in there. If after 72 hours no vetoes appear, I have no objections on adding this specific KIP as it shouldn't have a big blast radius of affectation. Best, On Wed, May 15, 2024 at 6:44 PM Josep Prat wrote: > Ah, I see Chris was f

Re: [VOTE] KIP-899: Allow producer and consumer clients to rebootstrap

Hi, Ivan, Thanks for the KIP. +1 Just a minor comment. Should we add metadata.recovery.strategy to the Streams and the newly introduced ShareConsumer (KIP-932) too? Jun On Wed, May 8, 2024 at 11:35 AM Manikumar wrote: > Thanks for the KIP. > > +1 (binding). > > On Wed, Apr 17, 2024 at 7:50 PM

Re: [DISCUSS] KIP-891: Running multiple versions of Connector plugins

Hey Chris, Thanks for your quick follow up. > But this risk is already present with > existing error cases, and I don't see anything that justifies changing > existing behavior with an invalid converter class, or diverging from it in > the case of invalid converter versions. The justification is

Re: [DISCUSS] KIP-1044: A proposal to change idempotent producer -- server implementation

+1 to the comment. > I still feel we are doing all of this only because of a few anti-pattern or misconfigured producers and not because we have “too many Producer”. I believe that implementing Producer heartbeat and remove short-lived PIDs from the cache if we didn’t receive heartbeat will be mo

[jira] [Created] (KAFKA-16773) Fix flaky QuorumControllerTest#testDelayedConfigurationOperations

Chia-Ping Tsai created KAFKA-16773: -- Summary: Fix flaky QuorumControllerTest#testDelayedConfigurationOperations Key: KAFKA-16773 URL: https://issues.apache.org/jira/browse/KAFKA-16773 Project: Kafka

[jira] [Created] (KAFKA-16774) fix flaky StreamThreadTest#shouldCloseAllTaskProducersOnCloseIfEosEnabled

Chia-Ping Tsai created KAFKA-16774: -- Summary: fix flaky StreamThreadTest#shouldCloseAllTaskProducersOnCloseIfEosEnabled Key: KAFKA-16774 URL: https://issues.apache.org/jira/browse/KAFKA-16774 Project

[jira] [Created] (KAFKA-16775) Fix flaky PlaintextAdminIntegrationTest#testCreateExistingTopicsThrowTopicExistsException

Chia-Ping Tsai created KAFKA-16775: -- Summary: Fix flaky PlaintextAdminIntegrationTest#testCreateExistingTopicsThrowTopicExistsException Key: KAFKA-16775 URL: https://issues.apache.org/jira/browse/KAFKA-16775

Re: [DISCUSS] Apache Kafka 3.8.0 release

Hi Josep, Would it be possible to sneak KIP-989 into 3.8? Just as with 1028, it's currently being voted on and has already received the requisite votes. The only thing holding it back is the 72 hour voting window. Vote thread here: https://lists.apache.org/thread/nhr65h4784z49jbsyt5nx8ys81q90k6s

Re: [VOTE] KIP-899: Allow producer and consumer clients to rebootstrap

Hi Ivan, Thanks for the KIP, +1 (binding) Regards, Rajini On Wed, May 15, 2024 at 6:15 PM Jun Rao wrote: > Hi, Ivan, > > Thanks for the KIP. +1 > > Just a minor comment. Should we add metadata.recovery.strategy to the > Streams and the newly introduced ShareConsumer (KIP-932) too? > > Jun >

Re: [DISCUSS] Apache Kafka 3.8.0 release

Hi Nick, If nobody comes up with concerns or pushback until the time of closing the vote, I think we can take it for 3.8. Best, - Josep Prat Open Source Engineering Director, aivenjosep.p...@aiven.io | +491715557497 | aiven.io Aiven Deutschland GmbH Alexanderufer 3-7, 10117 Be

Re: [VOTE] KIP-899: Allow producer and consumer clients to rebootstrap

Hi all, It seems we can close the vote in favor of the KIP: - Binding +1s: Chris Egerton, Manikumar Reddy, Jun Rao, Rajini Sivaram. - Non-binding +1s: Andrew Schofield, Omnia Ibrahim. - No -1s. Thank you all for your votes! I'll do the necessary changes to the KIPs page and proceed with the impl

Re: [VOTE] KIP-899: Allow producer and consumer clients to rebootstrap

Hi Jun, Thank you for you comment. I was thinking that this `metadata.recovery.strategy` could be passed to the relevant consumer in streams using the `restore.consumer.` prefix. I that what you meant or I misunderstand? As for the KIP-932, I'll have a closer look. Ivan On Wed, May 15, 2024,

[jira] [Created] (KAFKA-16776) Builds flaky-failing with leaked client-metrics-reaper thread

Greg Harris created KAFKA-16776: --- Summary: Builds flaky-failing with leaked client-metrics-reaper thread Key: KAFKA-16776 URL: https://issues.apache.org/jira/browse/KAFKA-16776 Project: Kafka

[jira] [Created] (KAFKA-16777) New consumer should throw NoOffsetForPartitionException on continuous poll zero if no reset strategy

Lianet Magrans created KAFKA-16777: -- Summary: New consumer should throw NoOffsetForPartitionException on continuous poll zero if no reset strategy Key: KAFKA-16777 URL: https://issues.apache.org/jira/browse/KAFKA

Re: [VOTE] KIP-932: Queues for Kafka

Hi, Andrew, Thanks for the update. Should we mark whether those metrics are standard/required for KIP-714? Jun On Tue, May 14, 2024 at 7:31 AM Andrew Schofield wrote: > Hi, > I have made a small update to the KIP as a result of testing the new > share consumer with client telemetry (KIP-714).

[jira] [Created] (KAFKA-16778) AsyncKafkaConsumer fetcher might occasionally try to fetch to a revoked partition

Philip Nee created KAFKA-16778: -- Summary: AsyncKafkaConsumer fetcher might occasionally try to fetch to a revoked partition Key: KAFKA-16778 URL: https://issues.apache.org/jira/browse/KAFKA-16778 Project

[jira] [Created] (KAFKA-16779) Kafka retains logs past specified retention

Nicholas Feinberg created KAFKA-16779: - Summary: Kafka retains logs past specified retention Key: KAFKA-16779 URL: https://issues.apache.org/jira/browse/KAFKA-16779 Project: Kafka Issue T

Re: [DISCUSS] KIP-924: customizable task assignment for Streams

Thanks Bruno! First, need to make one quick fix to what I said in the previous email -- the new rackId() getter will be added to KafkaStreamsState, not KafkaStreamsApplication (The KIP is correct, but what I put in the email was not) U1. I would actually prefer to keep the constructors as is, for

Re: [VOTE] KIP-899: Allow producer and consumer clients to rebootstrap

Hi, Ivan, You are right. StreamsConfigs can take all existing consumer configs, with or without prefixes. So, we don't need to add the new config to StreamsConfig explicitly. For KIP-932, it says for each new consumer config, we need to determine whether it should be added to ShareConsumer config

Re: [DISCUSS] KIP-989: RocksDB Iterator Metrics

One quick thing -- can you update the title of this KIP to reflect the decision to implement these metrics for all state stores implementations rather than just RocksDB? On Tue, May 14, 2024 at 1:36 PM Nick Telford wrote: > Woops! Thanks for the catch Lucas. Given this was just a typo, I don't

Re: [DISCUSS] KIP-1035: StateStore managed changelog offsets

103: I like the idea of immediately deprecating #managesOffsets and aiming to make offset management mandatory in the long run. I assume we would also log a warning for any custom stores that return "false" from this method to encourage custom store implementations to start doing so? My only questi

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #2907

See

Re: [DISCUSS] Apache Kafka 3.7.1 release

Hi Igor, Thanks for volunteering! +1 Luke On Wed, May 15, 2024 at 11:15 PM Mickael Maison wrote: > Hi Igor, > > Thanks for volunteering, +1 > > Mickael > > On Thu, Apr 25, 2024 at 11:09 AM Igor Soarez wrote: > > > > Hi everyone, > > > > I'd like to volunteer to be the release manager for a 3.

Re: [DISCUSS] Apache Kafka 3.8.0 release

Hi all, We are now officially past the KIP freeze deadline. KIPs that are approved after this point in time shouldn't be adopted in the 3.8.x release (except the 2 already mentioned KIPS 989 and 1028 assuming no vetoes occur). Reminder of the upcoming deadlines: - Feature freeze is on May 29th -