[DISCUSS] Docker Artifact Generation during Release

Hi Kafka Devs, In Kafka 3.7 we added the JVM Docker images in the release process. Then, in 3.8 we added the Native ones. After having done 4 release candidates for 3.8 I collected some experience on the process and I want to start a discussion on how we can optimize this. Currently, for each imag

[jira] [Resolved] (KAFKA-17188) LoginManager ctor might throw an exception causing login and loginCallbackHandler not being closed properly

[ https://issues.apache.org/jira/browse/KAFKA-17188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajini Sivaram resolved KAFKA-17188. Fix Version/s: 4.0.0 Reviewer: Rajini Sivaram Resolution: Fixed > LoginMan

[PR] KIP-848 is preview in 3.8 [kafka-site]

dajac opened a new pull request, #619: URL: https://github.com/apache/kafka-site/pull/619 We wanted to graduate KIP-848 to preview in 3.8. This patch fixes the blog post accordingly. Sorry for the confusion. I was on PTO until yesterday so I couldn't this feedback earlier. -- This

Re: [PR] KIP-848 is preview in 3.8 [kafka-site]

dajac commented on PR #619: URL: https://github.com/apache/kafka-site/pull/619#issuecomment-2259926736 @jlprat Could you please take a look? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the sp

Re: [PR] KIP-848 is preview in 3.8 [kafka-site]

jlprat commented on code in PR #619: URL: https://github.com/apache/kafka-site/pull/619#discussion_r1698121868 ## blog.html: ## @@ -43,8 +43,8 @@ Up until now, only the default compression level was used by Apache Kafka. From this version on, a configurati

Re: [PR] KIP-848 is preview in 3.8 [kafka-site]

jlprat commented on PR #619: URL: https://github.com/apache/kafka-site/pull/619#issuecomment-2259982912 > Sorry for the confusion. I was on PTO until yesterday so I couldn't this feedback earlier. Don't worry @dajac -- This is an automated message from the Apache Git Service. To r

Re: [PR] KIP-848 is preview in 3.8 [kafka-site]

dajac commented on code in PR #619: URL: https://github.com/apache/kafka-site/pull/619#discussion_r1698144860 ## blog.html: ## @@ -43,8 +43,8 @@ Up until now, only the default compression level was used by Apache Kafka. From this version on, a configuratio

Re: [PR] KIP-848 is preview in 3.8 [kafka-site]

jlprat commented on code in PR #619: URL: https://github.com/apache/kafka-site/pull/619#discussion_r1698150290 ## blog.html: ## @@ -43,8 +43,8 @@ Up until now, only the default compression level was used by Apache Kafka. From this version on, a configurati

[DISCUSS] Graduation steps for Features

Hi Kafka devs, Lately we started using "early access", "production ready" and also "preview" to determine the grade of "production readiness" of the features we deliver to our community. However, as far as I know, there is no official definition from the Apache Kafka side on which are the graduati

Re: [PR] KIP-848 is preview in 3.8 [kafka-site]

dajac commented on code in PR #619: URL: https://github.com/apache/kafka-site/pull/619#discussion_r1698164992 ## blog.html: ## @@ -43,8 +43,8 @@ Up until now, only the default compression level was used by Apache Kafka. From this version on, a configuratio

Re: [PR] KIP-848 is preview in 3.8 [kafka-site]

dajac merged PR #619: URL: https://github.com/apache/kafka-site/pull/619 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.or

Re: [DISCUSS] Graduation steps for Features

Also as part of this discussion I would like to flag that we need to be able to know how we can flag this properly so it's known for the Release Manager. For example, a KIP is approved, the Jira associated with it is being worked on. Release happens, Jira is still open, how can we flag that this KI

Re: [DISCUSS] Graduation steps for Features

Hi Josep, I think it’s high time that this was tackled. I suggest that it would be best handled as a KIP because then we have a document which can be discussed and improved, followed by a formal vote. A standard set of terms with agreed meanings would be very helpful for some of the larger KIPs wh

Re: [DISCUSS] Graduation steps for Features

Hi Andrew, I can definitely write a KIP, but before doing so I'd like to gather some feedback from the community around these steps and how they are perceived by different groups of people. On Wed, Jul 31, 2024 at 11:50 AM Andrew Schofield wrote: > Hi Josep, > I think it’s high time that this wa

[jira] [Created] (KAFKA-17225) Refactor consumer membership managers

Andrew Schofield created KAFKA-17225: Summary: Refactor consumer membership managers Key: KAFKA-17225 URL: https://issues.apache.org/jira/browse/KAFKA-17225 Project: Kafka Issue Type: Sub

[jira] [Created] (KAFKA-17226) Refactor pipeline to generate both docker artifacts

Vedarth Sharma created KAFKA-17226: -- Summary: Refactor pipeline to generate both docker artifacts Key: KAFKA-17226 URL: https://issues.apache.org/jira/browse/KAFKA-17226 Project: Kafka Issue

Re: [DISCUSS] Docker Artifact Generation during Release

Hi Josep, I agree with the suggestions. We can consider updating the pipelines to generate both artifacts from a single job and make the pipelines driven by RC number. I have created this jira for tracking the same https://issues.apache.org/jira/browse/KAFKA-17226 Thanks and regards, Vedarth On

Re: [DISCUSS] Docker Artifact Generation during Release

Thanks Vedarth! On Wed, Jul 31, 2024 at 12:24 PM Vedarth Sharma wrote: > Hi Josep, > > I agree with the suggestions. We can consider updating the pipelines to > generate both artifacts from a single job and make the pipelines driven by > RC number. > I have created this jira for tracking the sam

[jira] [Created] (KAFKA-17227) Apache Kafka 3.8.0 /tmp exec permission

Francisco Martinez created KAFKA-17227: -- Summary: Apache Kafka 3.8.0 /tmp exec permission Key: KAFKA-17227 URL: https://issues.apache.org/jira/browse/KAFKA-17227 Project: Kafka Issue Typ

[jira] [Created] (KAFKA-17228) Static member using new protocol should always replace the one using the old protocol

Dongnuo Lyu created KAFKA-17228: --- Summary: Static member using new protocol should always replace the one using the old protocol Key: KAFKA-17228 URL: https://issues.apache.org/jira/browse/KAFKA-17228 P

Re: [DISCUSS] Graduation steps for Features

Hi Josep, Thanks for starting the discussion. We used Early Access, Preview and GA (or Production Ready) for KIP-848 and I find it pretty nice. We could add the tentative release plan to the KIP's header and it could be used as the source of truth. Best, David On Wed, Jul 31, 2024 at 11:53 AM J

Re: [DISCUSS] Graduation steps for Features

Hi David, One of the problems I see is that the KIP index page has a 1-to-many relationship between KIP and release. I guess we might want to turn this to a many-to-many qualified relationship. Otherwise it might be complicated for the release manager or the KIP driver(s) to keep the Release Plan

Re: [DISCUSS] Graduation steps for Features

We could also use a hierarchie: KIP Parent Jira > Milestones Jiras > Tasks. Best, David On Wed, Jul 31, 2024 at 4:22 PM Josep Prat wrote: > Hi David, > > One of the problems I see is that the KIP index page has a 1-to-many > relationship between KIP and release. I guess we might want to turn th

Re: [DISCUSS] KIP-1016 Make MM2 heartbeats topic name configurable

Hi All, *Viktor * I was wrong sorry, Chris is right we have the per replication flow configuration for free. *Chris, regarding your comments.* *>IMO we don't need the "default." prefix* Sure I can do that, my intention with that was to make it clear for the users, that this property is not compat

[jira] [Created] (KAFKA-17229) Multiple punctuators that together exceed the transaction timeout cause ProducerFencedException

Trevan Richins created KAFKA-17229: -- Summary: Multiple punctuators that together exceed the transaction timeout cause ProducerFencedException Key: KAFKA-17229 URL: https://issues.apache.org/jira/browse/KAFKA-1722

Re: [DISCUSS] Graduation steps for Features

That sounds like a good idea. It will definitely clarify upfront what it means "Early Access", "Preview" and "Production Ready" for each KIP. I was looking at JIra and it seems we only have a 2 level hierarchy: Task(different types of) and Sub-Task. So we would need to change this. I was also think

Re: [DISCUSS] Graduation steps for Features

Hi Josep, Thanks for bringing up this discussion. My impression was that "preview" was earlier than early access. In my mind, "preview" somewhat implies that the feature could be incomplete or partial. For example, if someone said their game was a "preview" I would expect kind of a demo. Earl

Re: [DISCUSS] Graduation steps for Features

Hi Colin, I'm not attached to any names, I'm happy to change them. But from what I can guess from KIP-848 the order is EA -> Preview -> GA Maybe I'm mistaken. This is why this discussion is important so we all agree on the number of steps, order and names. I know finding a universal solution is

Re: [VOTE] KIP-877: Mechanism for plugins and connectors to register metrics

Hi, Bumping this thread to get some more votes and/or feedback. As I restarted the vote on June 10 after major changes, I'm only counting votes since then. So we have 1 binding (Chris) and 1 non-binding (Hector) votes. Thanks, Mickael On Mon, Jul 8, 2024 at 8:40 PM Hector Geraldino (BLOOMBERG/

[jira] [Created] (KAFKA-17230) Kafka consumer client doesn't report node request-latency metrics

Apoorv Mittal created KAFKA-17230: - Summary: Kafka consumer client doesn't report node request-latency metrics Key: KAFKA-17230 URL: https://issues.apache.org/jira/browse/KAFKA-17230 Project: Kafka

Re: [DISCUSS] Graduation steps for Features

Hi Josep, Thanks for starting this conversation! Yes, for KIPs that span multiple releases, clearly defining the KIP’s state for each release is very important. Perhaps the KIP template could be updated to include a “Release Plan” section that includes entries that include: - Version: future A

Re: [DISCUSS] Graduation steps for Features

I'd be hesitant to add too much to the KIP template. We should be careful to keep the barrier to entry low for new users. On Wed, Jul 31, 2024, 13:24 Kirk True wrote: > Hi Josep, > > Thanks for starting this conversation! > > Yes, for KIPs that span multiple releases, clearly defining the KIP’s

Re: [DISCUSS] KIP-512: make Record Headers available in onAcknowledgement

Hi Lianet, > LM1: Main advantage would be not having to add a header for calculating latency I see your points. I want to share our use case for latency measurements. We have a pipeline: `MySQL -> Debezium -> KafkaA -> Flink -> KafkaB`. In KIP-512, I simplified it to measure `Flink Sink -> KafkaB

[jira] [Created] (KAFKA-17231) Sn

Andrew Schofield created KAFKA-17231: Summary: Sn Key: KAFKA-17231 URL: https://issues.apache.org/jira/browse/KAFKA-17231 Project: Kafka Issue Type: Sub-task Reporter: Andrew

Jenkins build is unstable: Kafka » Kafka Branch Builder » 3.8 #77

See

[jira] [Created] (KAFKA-17232) MirrorCheckpointConnector does not generate task configs if initial consumer group load times out

Greg Harris created KAFKA-17232: --- Summary: MirrorCheckpointConnector does not generate task configs if initial consumer group load times out Key: KAFKA-17232 URL: https://issues.apache.org/jira/browse/KAFKA-17232

Re: [DISCUSS] Graduation steps for Features

Hi, I would at least like to see agreement on the various stages. I’m used to Early Access, then Preview, and finally General Availability, but I notice that at least one other person on this thread had the order different. Also, General Availability sounds a bit wrong for an open-source project, s

Re: [DISCUSS] Graduation steps for Features

Hi, I think we can keep the KIP template as is and have the extra process in the Jira area. EA, Preview and Production Ready (I also kind of like this better) feel to me more project management while the KIP document itself is more about the nature of the changes and describing the end result. Ex

[jira] [Created] (KAFKA-17233) MirrorCheckpointConnector should use batched listConsumerGroupOffsets

Greg Harris created KAFKA-17233: --- Summary: MirrorCheckpointConnector should use batched listConsumerGroupOffsets Key: KAFKA-17233 URL: https://issues.apache.org/jira/browse/KAFKA-17233 Project: Kafka

Re: New release branch 3.9

Hi Colin, Would it be alright if I cherry-picked https://github.com/apache/kafka/pull/16678 onto 3.9? This isn't a regression but it's a low-risk fix that I'd like to get into the next release if possible. Cheers, Chris On Wed, Jul 31, 2024 at 12:29 AM Ismael Juma wrote: > I would recommend a

Jenkins build is unstable: Kafka » Kafka Branch Builder » trunk #3156

See

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 3.9 #7

See

Re: [DISCUSS] KIP-877: Mechanism for plugins and connectors to register metrics

Hi Mickael, I took another look at this KIP (prompted by your bumping of the vote thread) and had a few more points: 1. `PluginMetrics.metricName()` says "Tags to uniquely identify the plugins are automatically added to the provided tags". This would seem to imply that either some tags may not be

[jira] [Created] (KAFKA-17234) Partition.tryCompleteDelayedRequests should not throw

Colin McCabe created KAFKA-17234: Summary: Partition.tryCompleteDelayedRequests should not throw Key: KAFKA-17234 URL: https://issues.apache.org/jira/browse/KAFKA-17234 Project: Kafka Issue

Re: [DISCUSS] Graduation steps for Features

Hi, thanks for starting this discussion. It's for sure useful. I don't care too much about the naming personally, it just should be well defined. I agree to Josep, that I don't think we would need to change the KIP template. A KIP is a design doc for the full feature, and should not concern

Re: [DISCUSS] KIP-512: make Record Headers available in onAcknowledgement

Hi Matthias, > (100) To me, Headers store "application metadata" of a record, but they are not "Kafka native" record metadata fair enough. > (200) extending `onComplete()` and pass in the Headers might be a good thing to do, too I agree it’s a nice feature to have. We can increase the scope for

Inquiry on Compact Topic Behavior in Kafka 3.7.0

Hi, dev community, We are currently working with Kafka 3.7.0 and have configured a topic with the following settings: log.cleanup.policy=compact segment.ms=12 (2 minutes) log.roll.hours (default 7 days) This topic is used for testing purposes and has a low message volume. Specifically, it co

Re: [DISCUSS] Graduation steps for Features

I only raised the idea of barrier to entry regarding the template. I did not state or even really imply that it should affect this initiative in general. Please do not feel obligated to discuss that concept further if we're all on the same page RE not adding to the template. On Wed, Jul 31, 2024,

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 3.9 #8

See

[jira] [Created] (KAFKA-17235) system test test_performance_service.py failed

Kuan Po Tseng created KAFKA-17235: - Summary: system test test_performance_service.py failed Key: KAFKA-17235 URL: https://issues.apache.org/jira/browse/KAFKA-17235 Project: Kafka Issue Type:

[jira] [Created] (KAFKA-17236) local log should be deleted when remote copy disabled and no remote logs

Luke Chen created KAFKA-17236: - Summary: local log should be deleted when remote copy disabled and no remote logs Key: KAFKA-17236 URL: https://issues.apache.org/jira/browse/KAFKA-17236 Project: Kafka