[DISCUSS] Docker Artifact Generation during Release

[Josep Prat]

Hi Kafka Devs,
In Kafka 3.7 we added the JVM Docker images in the release process. Then,
in 3.8 we added the Native ones.
After having done 4 release candidates for 3.8 I collected some experience
on the process and I want to start a discussion on how we can optimize this.

Currently, for each image we need to first build the image and then once
it's finished, promote it as RC. This needs to be done independently for
the JVM and for the Native images. The promotion workflow needs 3 different
fields, 2 of which are really linked (type of the image and the name of it).
An example run would be:
- type: jvm
- RC image: apache/kafka:3.8.0-rc3
- Kafka url:
https://home.apache.org/~jlprat/kafka-3.8.0-rc3/kafka_2.13-3.8.0.tgz
While for native it would be:
- type: native
- RC image: apache/kafka-native:3.8.0-rc3
- Kafka url:
https://home.apache.org/~jlprat/kafka-3.8.0-rc3/kafka_2.13-3.8.0.tgz

It is really easy to make a mistake and select a native one with the wrong
image name (or vice versa).

The changes that I think would make sense to perform to the workflows are:
- Have a combined flow for generating both images at the same time (we
always need to generate both)
-- This should internally call the 2 workflows
- Have a combined flow for promoting images to RC
-- This should take then  the RC number instead of the full name as the
name of the image is fixed

A note on the process as well, the generation of the Native image takes
almost an hour, so for future release managers, take some time when you
generate them.

Any thoughts about it?

Best,
-- 
[image: Aiven] 

*Josep Prat*
Open Source Engineering Director, *Aiven*
josep.p...@aiven.io   |   +491715557497
aiven.io    |   
     
*Aiven Deutschland GmbH*
Alexanderufer 3-7, 10117 Berlin
Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen,
Anna Richardson, Kenneth Chen
Amtsgericht Charlottenburg, HRB 209739 B

[jira] [Resolved] (KAFKA-17188) LoginManager ctor might throw an exception causing login and loginCallbackHandler not being closed properly

[Rajini Sivaram (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-17188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajini Sivaram resolved KAFKA-17188.

Fix Version/s: 4.0.0
 Reviewer: Rajini Sivaram
   Resolution: Fixed

> LoginManager ctor might throw an exception causing login and 
> loginCallbackHandler not being closed properly
> ---
>
> Key: KAFKA-17188
> URL: https://issues.apache.org/jira/browse/KAFKA-17188
> Project: Kafka
>  Issue Type: Bug
>  Components: security
>Reporter: Philip Nee
>Assignee: Philip Nee
>Priority: Major
> Fix For: 4.0.0
>
>
> When using MDS login, loginManager.login() may throw an exception causing 
> login and loginCallbackHandler not being closed properly.  We should catch 
> exception and close login and loginCallbackHandler.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[PR] KIP-848 is preview in 3.8 [kafka-site]

[via GitHub]

dajac opened a new pull request, #619:
URL: https://github.com/apache/kafka-site/pull/619

   We wanted to graduate KIP-848 to preview in 3.8. This patch fixes the blog 
post accordingly.
   
   Sorry for the confusion. I was on PTO until yesterday so I couldn't this 
feedback earlier.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] KIP-848 is preview in 3.8 [kafka-site]

[via GitHub]

dajac commented on PR #619:
URL: https://github.com/apache/kafka-site/pull/619#issuecomment-2259926736

   @jlprat Could you please take a look?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] KIP-848 is preview in 3.8 [kafka-site]

[via GitHub]

jlprat commented on code in PR #619:
URL: https://github.com/apache/kafka-site/pull/619#discussion_r1698121868


##
blog.html:
##
@@ -43,8 +43,8 @@ 
 Up until now, only the default compression level was used 
by Apache Kafka. From this version on, a configuration mechanism to specify 
compression level is included. See https://cwiki.apache.org/confluence/display/KAFKA/KIP-390%3A+Support+Compression+Level";>KIP-390
 for more details.
 
 
-In the last release, 3.7, https://cwiki.apache.org/confluence/display/KAFKA/KIP-848%3A+The+Next+Generation+of+the+Consumer+Rebalance+Protocol";>KIP-848
 The Next Generation of the Consumer Rebalance Protocol was made available 
as early access. This version includes numerous bug
-fixes and the community is encouraged to test and provide 
feedback. https://cwiki.apache.org/confluence/display/KAFKA/The+Next+Generation+of+the+Consumer+Rebalance+Protocol+%28KIP-848%29+-+Early+Access+Release+Notes";>See
 the early access release notes for more information.
+https://cwiki.apache.org/confluence/display/KAFKA/KIP-848%3A+The+Next+Generation+of+the+Consumer+Rebalance+Protocol";>KIP-848
 The Next Generation of the Consumer Rebalance Protocol is available as 
preview in 3.8. This version includes numerous bug

Review Comment:
   Do we have the concept of `preview` in Kafka?
   I know we had several discussions in previous PRs regarding "early access" 
and "production ready" not being really defined 
(https://github.com/apache/kafka-site/pull/614#discussion_r1686561076 for 
example)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] KIP-848 is preview in 3.8 [kafka-site]

[via GitHub]

jlprat commented on PR #619:
URL: https://github.com/apache/kafka-site/pull/619#issuecomment-2259982912

   > Sorry for the confusion. I was on PTO until yesterday so I couldn't this 
feedback earlier.
   
   Don't worry @dajac 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] KIP-848 is preview in 3.8 [kafka-site]

[via GitHub]

dajac commented on code in PR #619:
URL: https://github.com/apache/kafka-site/pull/619#discussion_r1698144860


##
blog.html:
##
@@ -43,8 +43,8 @@ 
 Up until now, only the default compression level was used 
by Apache Kafka. From this version on, a configuration mechanism to specify 
compression level is included. See https://cwiki.apache.org/confluence/display/KAFKA/KIP-390%3A+Support+Compression+Level";>KIP-390
 for more details.
 
 
-In the last release, 3.7, https://cwiki.apache.org/confluence/display/KAFKA/KIP-848%3A+The+Next+Generation+of+the+Consumer+Rebalance+Protocol";>KIP-848
 The Next Generation of the Consumer Rebalance Protocol was made available 
as early access. This version includes numerous bug
-fixes and the community is encouraged to test and provide 
feedback. https://cwiki.apache.org/confluence/display/KAFKA/The+Next+Generation+of+the+Consumer+Rebalance+Protocol+%28KIP-848%29+-+Early+Access+Release+Notes";>See
 the early access release notes for more information.
+https://cwiki.apache.org/confluence/display/KAFKA/KIP-848%3A+The+Next+Generation+of+the+Consumer+Rebalance+Protocol";>KIP-848
 The Next Generation of the Consumer Rebalance Protocol is available as 
preview in 3.8. This version includes numerous bug

Review Comment:
   I think that we don't have any official graduation terminology for new 
features in Apache Kafka. It may be worth to discuss it on the mailing list to 
determine them. That being said, I think that we used Preview in the past too.
   
   From a previous blog post:
   
   > KIP-796 is a long-term project that will be extended with new query types 
in future releases. As of Apache Kafka 3.2.0, IQv2 is in preview. The public 
documentation site has not been updated, and the interfaces of IQv2 are marked 
as @Evolving (meaning that they may break compatibility in minor releases 
without a deprecation period if preview users find significant flaws in the 
current API). A future release will remove the @Evolving annotation and 
designate IQv2 as stable.
   
   I would like to stress in this release that KIP-848 has made significant 
progress since the last release from a quality and stability point of view. Our 
early access was really early from this point of view.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] KIP-848 is preview in 3.8 [kafka-site]

[via GitHub]

jlprat commented on code in PR #619:
URL: https://github.com/apache/kafka-site/pull/619#discussion_r1698150290


##
blog.html:
##
@@ -43,8 +43,8 @@ 
 Up until now, only the default compression level was used 
by Apache Kafka. From this version on, a configuration mechanism to specify 
compression level is included. See https://cwiki.apache.org/confluence/display/KAFKA/KIP-390%3A+Support+Compression+Level";>KIP-390
 for more details.
 
 
-In the last release, 3.7, https://cwiki.apache.org/confluence/display/KAFKA/KIP-848%3A+The+Next+Generation+of+the+Consumer+Rebalance+Protocol";>KIP-848
 The Next Generation of the Consumer Rebalance Protocol was made available 
as early access. This version includes numerous bug
-fixes and the community is encouraged to test and provide 
feedback. https://cwiki.apache.org/confluence/display/KAFKA/The+Next+Generation+of+the+Consumer+Rebalance+Protocol+%28KIP-848%29+-+Early+Access+Release+Notes";>See
 the early access release notes for more information.
+https://cwiki.apache.org/confluence/display/KAFKA/KIP-848%3A+The+Next+Generation+of+the+Consumer+Rebalance+Protocol";>KIP-848
 The Next Generation of the Consumer Rebalance Protocol is available as 
preview in 3.8. This version includes numerous bug

Review Comment:
   Ah this is from the "old" blog posts. From 3.5.0 onwards I couldn't find any 
reference to "Preview".
   I know what you mean with wanting to flag that this KIP moved "upwards" in 
terms of production readiness. Let's accept this here and I'll open a discuss 
thread in the mailing list to clarify our feature graduation steps.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[DISCUSS] Graduation steps for Features

[Josep Prat]

Hi Kafka devs,

Lately we started using "early access", "production ready" and also
"preview" to determine the grade of "production readiness" of the features
we deliver to our community.
However, as far as I know, there is no official definition from the Apache
Kafka side on which are the graduation steps for features and what type of
"guarantees" each of these offer.

I think we should agree on which terms we should use and what each of these
exactly mean in terms of reliability. So far it seems we have this
graduation steps:
- Early Access: Feature is just complete but not yet fully polished and
maybe not used in production in many environments
- Preview: Feature was early access before and it underwent at least a
cycle of improvements and fixes and it's used in some production
environments maybe
- Production ready: Feature is officially released and it fulfills the
expected initial needs

Note that we don't offer any guarantees or SLA/SLO in the classical term.

Is this something we can agree on? What do those terms mean to you? Do we
need more steps? Or do we need less steps?

Best,
-- 
[image: Aiven] 

*Josep Prat*
Open Source Engineering Director, *Aiven*
josep.p...@aiven.io   |   +491715557497
aiven.io    |   
     
*Aiven Deutschland GmbH*
Alexanderufer 3-7, 10117 Berlin
Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen,
Anna Richardson, Kenneth Chen
Amtsgericht Charlottenburg, HRB 209739 B

Re: [PR] KIP-848 is preview in 3.8 [kafka-site]

[via GitHub]

dajac commented on code in PR #619:
URL: https://github.com/apache/kafka-site/pull/619#discussion_r1698164992


##
blog.html:
##
@@ -43,8 +43,8 @@ 
 Up until now, only the default compression level was used 
by Apache Kafka. From this version on, a configuration mechanism to specify 
compression level is included. See https://cwiki.apache.org/confluence/display/KAFKA/KIP-390%3A+Support+Compression+Level";>KIP-390
 for more details.
 
 
-In the last release, 3.7, https://cwiki.apache.org/confluence/display/KAFKA/KIP-848%3A+The+Next+Generation+of+the+Consumer+Rebalance+Protocol";>KIP-848
 The Next Generation of the Consumer Rebalance Protocol was made available 
as early access. This version includes numerous bug
-fixes and the community is encouraged to test and provide 
feedback. https://cwiki.apache.org/confluence/display/KAFKA/The+Next+Generation+of+the+Consumer+Rebalance+Protocol+%28KIP-848%29+-+Early+Access+Release+Notes";>See
 the early access release notes for more information.
+https://cwiki.apache.org/confluence/display/KAFKA/KIP-848%3A+The+Next+Generation+of+the+Consumer+Rebalance+Protocol";>KIP-848
 The Next Generation of the Consumer Rebalance Protocol is available as 
preview in 3.8. This version includes numerous bug

Review Comment:
   Thanks!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] KIP-848 is preview in 3.8 [kafka-site]

[via GitHub]

dajac merged PR #619:
URL: https://github.com/apache/kafka-site/pull/619


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [DISCUSS] Graduation steps for Features

[Josep Prat]

Also as part of this discussion I would like to flag that we need to be
able to know how we can flag this properly so it's known for the Release
Manager.
For example, a KIP is approved, the Jira associated with it is being worked
on. Release happens, Jira is still open, how can we flag that this KIP is
in early access, or preview?

Best,

On Wed, Jul 31, 2024 at 11:03 AM Josep Prat  wrote:

> Hi Kafka devs,
>
> Lately we started using "early access", "production ready" and also
> "preview" to determine the grade of "production readiness" of the features
> we deliver to our community.
> However, as far as I know, there is no official definition from the Apache
> Kafka side on which are the graduation steps for features and what type of
> "guarantees" each of these offer.
>
> I think we should agree on which terms we should use and what each of
> these exactly mean in terms of reliability. So far it seems we have this
> graduation steps:
> - Early Access: Feature is just complete but not yet fully polished and
> maybe not used in production in many environments
> - Preview: Feature was early access before and it underwent at least a
> cycle of improvements and fixes and it's used in some production
> environments maybe
> - Production ready: Feature is officially released and it fulfills the
> expected initial needs
>
> Note that we don't offer any guarantees or SLA/SLO in the classical term.
>
> Is this something we can agree on? What do those terms mean to you? Do we
> need more steps? Or do we need less steps?
>
> Best,
> --
> [image: Aiven] 
>
> *Josep Prat*
> Open Source Engineering Director, *Aiven*
> josep.p...@aiven.io   |   +491715557497
> aiven.io    |
> 
>    
> *Aiven Deutschland GmbH*
> Alexanderufer 3-7, 10117 Berlin
> Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen,
> Anna Richardson, Kenneth Chen
> Amtsgericht Charlottenburg, HRB 209739 B
>


-- 
[image: Aiven] 

*Josep Prat*
Open Source Engineering Director, *Aiven*
josep.p...@aiven.io   |   +491715557497
aiven.io    |   
     
*Aiven Deutschland GmbH*
Alexanderufer 3-7, 10117 Berlin
Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen,
Anna Richardson, Kenneth Chen
Amtsgericht Charlottenburg, HRB 209739 B

Re: [DISCUSS] Graduation steps for Features

[Andrew Schofield]

Hi Josep,
I think it’s high time that this was tackled. I suggest that it would be best
handled as a KIP because then we have a document which can be discussed
and improved, followed by a formal vote.

A standard set of terms with agreed meanings would be very helpful for
some of the larger KIPs which take many releases to be properly ready for
prime time. Most KIPs don’t need this, but a handful definitely do.

Personally, I like the sequence that KIP-848 has taken, moving from Early
Access, to Preview, and finally complete. I intend to follow the same sequence
for KIP-932.

Thanks,
Andrew

> On 31 Jul 2024, at 10:15, Josep Prat  wrote:
>
> Also as part of this discussion I would like to flag that we need to be
> able to know how we can flag this properly so it's known for the Release
> Manager.
> For example, a KIP is approved, the Jira associated with it is being worked
> on. Release happens, Jira is still open, how can we flag that this KIP is
> in early access, or preview?
>
> Best,
>
> On Wed, Jul 31, 2024 at 11:03 AM Josep Prat  wrote:
>
>> Hi Kafka devs,
>>
>> Lately we started using "early access", "production ready" and also
>> "preview" to determine the grade of "production readiness" of the features
>> we deliver to our community.
>> However, as far as I know, there is no official definition from the Apache
>> Kafka side on which are the graduation steps for features and what type of
>> "guarantees" each of these offer.
>>
>> I think we should agree on which terms we should use and what each of
>> these exactly mean in terms of reliability. So far it seems we have this
>> graduation steps:
>> - Early Access: Feature is just complete but not yet fully polished and
>> maybe not used in production in many environments
>> - Preview: Feature was early access before and it underwent at least a
>> cycle of improvements and fixes and it's used in some production
>> environments maybe
>> - Production ready: Feature is officially released and it fulfills the
>> expected initial needs
>>
>> Note that we don't offer any guarantees or SLA/SLO in the classical term.
>>
>> Is this something we can agree on? What do those terms mean to you? Do we
>> need more steps? Or do we need less steps?
>>
>> Best,
>> --
>> [image: Aiven] 
>>
>> *Josep Prat*
>> Open Source Engineering Director, *Aiven*
>> josep.p...@aiven.io   |   +491715557497
>> aiven.io    |
>> 
>>    
>> *Aiven Deutschland GmbH*
>> Alexanderufer 3-7, 10117 Berlin
>> Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen,
>> Anna Richardson, Kenneth Chen
>> Amtsgericht Charlottenburg, HRB 209739 B
>>
>
>
> --
> [image: Aiven] 
>
> *Josep Prat*
> Open Source Engineering Director, *Aiven*
> josep.p...@aiven.io   |   +491715557497
> aiven.io    |   
>     
> *Aiven Deutschland GmbH*
> Alexanderufer 3-7, 10117 Berlin
> Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen,
> Anna Richardson, Kenneth Chen
> Amtsgericht Charlottenburg, HRB 209739 B

Re: [DISCUSS] Graduation steps for Features

[Josep Prat]

Hi Andrew,
I can definitely write a KIP, but before doing so I'd like to gather some
feedback from the community around these steps and how they are perceived
by different groups of people.

On Wed, Jul 31, 2024 at 11:50 AM Andrew Schofield 
wrote:

> Hi Josep,
> I think it’s high time that this was tackled. I suggest that it would be
> best
> handled as a KIP because then we have a document which can be discussed
> and improved, followed by a formal vote.
>
> A standard set of terms with agreed meanings would be very helpful for
> some of the larger KIPs which take many releases to be properly ready for
> prime time. Most KIPs don’t need this, but a handful definitely do.
>
> Personally, I like the sequence that KIP-848 has taken, moving from Early
> Access, to Preview, and finally complete. I intend to follow the same
> sequence
> for KIP-932.
>
> Thanks,
> Andrew
>
> > On 31 Jul 2024, at 10:15, Josep Prat 
> wrote:
> >
> > Also as part of this discussion I would like to flag that we need to be
> > able to know how we can flag this properly so it's known for the Release
> > Manager.
> > For example, a KIP is approved, the Jira associated with it is being
> worked
> > on. Release happens, Jira is still open, how can we flag that this KIP is
> > in early access, or preview?
> >
> > Best,
> >
> > On Wed, Jul 31, 2024 at 11:03 AM Josep Prat  wrote:
> >
> >> Hi Kafka devs,
> >>
> >> Lately we started using "early access", "production ready" and also
> >> "preview" to determine the grade of "production readiness" of the
> features
> >> we deliver to our community.
> >> However, as far as I know, there is no official definition from the
> Apache
> >> Kafka side on which are the graduation steps for features and what type
> of
> >> "guarantees" each of these offer.
> >>
> >> I think we should agree on which terms we should use and what each of
> >> these exactly mean in terms of reliability. So far it seems we have this
> >> graduation steps:
> >> - Early Access: Feature is just complete but not yet fully polished and
> >> maybe not used in production in many environments
> >> - Preview: Feature was early access before and it underwent at least a
> >> cycle of improvements and fixes and it's used in some production
> >> environments maybe
> >> - Production ready: Feature is officially released and it fulfills the
> >> expected initial needs
> >>
> >> Note that we don't offer any guarantees or SLA/SLO in the classical
> term.
> >>
> >> Is this something we can agree on? What do those terms mean to you? Do
> we
> >> need more steps? Or do we need less steps?
> >>
> >> Best,
> >> --
> >> [image: Aiven] 
> >>
> >> *Josep Prat*
> >> Open Source Engineering Director, *Aiven*
> >> josep.p...@aiven.io   |   +491715557497
> >> aiven.io    |
> >> 
> >>    <
> https://twitter.com/aiven_io>
> >> *Aiven Deutschland GmbH*
> >> Alexanderufer 3-7, 10117 Berlin
> >> Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen,
> >> Anna Richardson, Kenneth Chen
> >> Amtsgericht Charlottenburg, HRB 209739 B
> >>
> >
> >
> > --
> > [image: Aiven] 
> >
> > *Josep Prat*
> > Open Source Engineering Director, *Aiven*
> > josep.p...@aiven.io   |   +491715557497
> > aiven.io    |   <
> https://www.facebook.com/aivencloud>
> >     <
> https://twitter.com/aiven_io>
> > *Aiven Deutschland GmbH*
> > Alexanderufer 3-7, 10117 Berlin
> > Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen,
> > Anna Richardson, Kenneth Chen
> > Amtsgericht Charlottenburg, HRB 209739 B
>
>

-- 
[image: Aiven] 

*Josep Prat*
Open Source Engineering Director, *Aiven*
josep.p...@aiven.io   |   +491715557497
aiven.io    |   
     
*Aiven Deutschland GmbH*
Alexanderufer 3-7, 10117 Berlin
Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen,
Anna Richardson, Kenneth Chen
Amtsgericht Charlottenburg, HRB 209739 B

[jira] [Created] (KAFKA-17225) Refactor consumer membership managers

[Andrew Schofield (Jira)]

Andrew Schofield created KAFKA-17225:


 Summary: Refactor consumer membership managers
 Key: KAFKA-17225
 URL: https://issues.apache.org/jira/browse/KAFKA-17225
 Project: Kafka
  Issue Type: Sub-task
Reporter: Andrew Schofield
Assignee: Andrew Schofield
 Fix For: 4.0.0


The initial drop of ShareMembershipManager contained a lot of code duplicated 
from MembershipManagerImpl. The plan was always to share as much code as 
possible between the membership managers for consumer groups and share groups. 
This issue refactors the membership managers so that almost all of the code is 
in common.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-17226) Refactor pipeline to generate both docker artifacts

[Vedarth Sharma (Jira)]

Vedarth Sharma created KAFKA-17226:
--

 Summary: Refactor pipeline to generate both docker artifacts
 Key: KAFKA-17226
 URL: https://issues.apache.org/jira/browse/KAFKA-17226
 Project: Kafka
  Issue Type: Sub-task
Reporter: Vedarth Sharma


We currently have two separate docker images which are built and released by 
two separate pipeline runs. We can merge them in a single workflow so that we 
won't need to generate them separately. Also we need to reduce the possibility 
of human error by allowing the RM to only input the RC number instead of the 
entire image name



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] Docker Artifact Generation during Release

[Vedarth Sharma]

Hi Josep,

I agree with the suggestions. We can consider updating the pipelines to
generate both artifacts from a single job and make the pipelines driven by
RC number.
I have created this jira for tracking the same
https://issues.apache.org/jira/browse/KAFKA-17226

Thanks and regards,
Vedarth

On Wed, Jul 31, 2024 at 12:59 PM Josep Prat 
wrote:

> Hi Kafka Devs,
> In Kafka 3.7 we added the JVM Docker images in the release process. Then,
> in 3.8 we added the Native ones.
> After having done 4 release candidates for 3.8 I collected some experience
> on the process and I want to start a discussion on how we can optimize
> this.
>
> Currently, for each image we need to first build the image and then once
> it's finished, promote it as RC. This needs to be done independently for
> the JVM and for the Native images. The promotion workflow needs 3 different
> fields, 2 of which are really linked (type of the image and the name of
> it).
> An example run would be:
> - type: jvm
> - RC image: apache/kafka:3.8.0-rc3
> - Kafka url:
> https://home.apache.org/~jlprat/kafka-3.8.0-rc3/kafka_2.13-3.8.0.tgz
> While for native it would be:
> - type: native
> - RC image: apache/kafka-native:3.8.0-rc3
> - Kafka url:
> https://home.apache.org/~jlprat/kafka-3.8.0-rc3/kafka_2.13-3.8.0.tgz
>
> It is really easy to make a mistake and select a native one with the wrong
> image name (or vice versa).
>
> The changes that I think would make sense to perform to the workflows are:
> - Have a combined flow for generating both images at the same time (we
> always need to generate both)
> -- This should internally call the 2 workflows
> - Have a combined flow for promoting images to RC
> -- This should take then  the RC number instead of the full name as the
> name of the image is fixed
>
> A note on the process as well, the generation of the Native image takes
> almost an hour, so for future release managers, take some time when you
> generate them.
>
> Any thoughts about it?
>
> Best,
> --
> [image: Aiven] 
>
> *Josep Prat*
> Open Source Engineering Director, *Aiven*
> josep.p...@aiven.io   |   +491715557497
> aiven.io    |    >
>      <
> https://twitter.com/aiven_io>
> *Aiven Deutschland GmbH*
> Alexanderufer 3-7, 10117 Berlin
> Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen,
> Anna Richardson, Kenneth Chen
> Amtsgericht Charlottenburg, HRB 209739 B
>

Re: [DISCUSS] Docker Artifact Generation during Release

[Josep Prat]

Thanks Vedarth!

On Wed, Jul 31, 2024 at 12:24 PM Vedarth Sharma 
wrote:

> Hi Josep,
>
> I agree with the suggestions. We can consider updating the pipelines to
> generate both artifacts from a single job and make the pipelines driven by
> RC number.
> I have created this jira for tracking the same
> https://issues.apache.org/jira/browse/KAFKA-17226
>
> Thanks and regards,
> Vedarth
>
> On Wed, Jul 31, 2024 at 12:59 PM Josep Prat 
> wrote:
>
> > Hi Kafka Devs,
> > In Kafka 3.7 we added the JVM Docker images in the release process. Then,
> > in 3.8 we added the Native ones.
> > After having done 4 release candidates for 3.8 I collected some
> experience
> > on the process and I want to start a discussion on how we can optimize
> > this.
> >
> > Currently, for each image we need to first build the image and then once
> > it's finished, promote it as RC. This needs to be done independently for
> > the JVM and for the Native images. The promotion workflow needs 3
> different
> > fields, 2 of which are really linked (type of the image and the name of
> > it).
> > An example run would be:
> > - type: jvm
> > - RC image: apache/kafka:3.8.0-rc3
> > - Kafka url:
> > https://home.apache.org/~jlprat/kafka-3.8.0-rc3/kafka_2.13-3.8.0.tgz
> > While for native it would be:
> > - type: native
> > - RC image: apache/kafka-native:3.8.0-rc3
> > - Kafka url:
> > https://home.apache.org/~jlprat/kafka-3.8.0-rc3/kafka_2.13-3.8.0.tgz
> >
> > It is really easy to make a mistake and select a native one with the
> wrong
> > image name (or vice versa).
> >
> > The changes that I think would make sense to perform to the workflows
> are:
> > - Have a combined flow for generating both images at the same time (we
> > always need to generate both)
> > -- This should internally call the 2 workflows
> > - Have a combined flow for promoting images to RC
> > -- This should take then  the RC number instead of the full name as the
> > name of the image is fixed
> >
> > A note on the process as well, the generation of the Native image takes
> > almost an hour, so for future release managers, take some time when you
> > generate them.
> >
> > Any thoughts about it?
> >
> > Best,
> > --
> > [image: Aiven] 
> >
> > *Josep Prat*
> > Open Source Engineering Director, *Aiven*
> > josep.p...@aiven.io   |   +491715557497
> > aiven.io    |   <
> https://www.facebook.com/aivencloud
> > >
> >      <
> > https://twitter.com/aiven_io>
> > *Aiven Deutschland GmbH*
> > Alexanderufer 3-7, 10117 Berlin
> > Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen,
> > Anna Richardson, Kenneth Chen
> > Amtsgericht Charlottenburg, HRB 209739 B
> >
>


-- 
[image: Aiven] 

*Josep Prat*
Open Source Engineering Director, *Aiven*
josep.p...@aiven.io   |   +491715557497
aiven.io    |   
     
*Aiven Deutschland GmbH*
Alexanderufer 3-7, 10117 Berlin
Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen,
Anna Richardson, Kenneth Chen
Amtsgericht Charlottenburg, HRB 209739 B

[jira] [Created] (KAFKA-17227) Apache Kafka 3.8.0 /tmp exec permission

[Francisco Martinez (Jira)]

Francisco Martinez created KAFKA-17227:
--

 Summary: Apache Kafka 3.8.0 /tmp exec permission
 Key: KAFKA-17227
 URL: https://issues.apache.org/jira/browse/KAFKA-17227
 Project: Kafka
  Issue Type: Bug
Affects Versions: 3.8.0
Reporter: Francisco Martinez
 Attachments: kafka_issue.png

I have just downloaded and installed new Apache Kafka version 3.8.0.

It does not work for me (version 3.7.1 works fine).

In SLES 15 SP5, i have configured /etc/fstab to do not have exec permission for 
the /tmp partition (noexec) (as suggested by the CIS benchmark).

Then the Kafka service does not start. Even the kafka-storage.sh script cannot 
be executed to create the Kafka storage in /tmp/kraft-combined-logs.

The error reported (in all cases) is exception java.lang.UnsatisfiedLinkError: 
/tmp/lib/libzstd-jni-1.5.6-3.so: failed to map segment from shared object. 
The error does not appear if i enable again the exec permission in /tmp (i.e. 
mount -o remount,exec /tmp).

It seems that the zstd-jni-1.5.6-3.jar library is tried to be loaded (even in 
the case the compression is disabled by default in producer.properties: 
compression.type=none). Inside the jar file there is for example 
lizstd-jni-1.5.6-3.so for amd64 architecture that is used by the jar, and for 
that purpose it is copied to /tmp. But if the /tmp does not have execution 
permissions, the Apache Kafka processes don't start.

Maybe the issue is in zstd-jni and has to be solved in zstd-jni, or maybe the 
library could be imported only when necessary (only in case the compression is 
used) to minimize the issue with zstd-jni.

Thanks and regards.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-17228) Static member using new protocol should always replace the one using the old protocol

[Dongnuo Lyu (Jira)]

Dongnuo Lyu created KAFKA-17228:
---

 Summary: Static member using new protocol should always replace 
the one using the old protocol
 Key: KAFKA-17228
 URL: https://issues.apache.org/jira/browse/KAFKA-17228
 Project: Kafka
  Issue Type: Bug
Reporter: Dongnuo Lyu
Assignee: Dongnuo Lyu


{color:#172b4d}In the old protocol, when a static consumer shuts down, it 
[won't send explicit LeaveGroup 
request|https://github.com/apache/kafka/blob/010ab19b724ae011e85686ce47320f4f85d9a11f/clients/src/main/java/org/apache/kafka/clients/consumer/internals/AbstractCoordinator.java#L1158-L1164].
 It's okay because the old protocol replaces the existing member whenever a new 
member with the same instance id joins.{color}

{color:#172b4d}However in the new protocol, we [requires the existing member to 
send leave 
group|https://github.com/apache/kafka/blob/trunk/group-coordinator/src/main/java/org/apache/kafka/coordinator/group/GroupMetadataManager.java#L2236-L2238]
 for a new static member to replace the existing one. The gap causes the 
upgraded new consumer unable to join the group in both online/offline 
upgrade.{color}

{color:#172b4d}We should make the static member using new protocol replace the 
static member using old protocol regardless of whether the latter has left the 
group.{color}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] Graduation steps for Features

[David Jacot]

Hi Josep,

Thanks for starting the discussion.

We used Early Access, Preview and GA (or Production Ready) for KIP-848 and
I find it pretty nice. We could add the tentative release plan to the KIP's
header and it could be used as the source of truth.

Best,
David

On Wed, Jul 31, 2024 at 11:53 AM Josep Prat 
wrote:

> Hi Andrew,
> I can definitely write a KIP, but before doing so I'd like to gather some
> feedback from the community around these steps and how they are perceived
> by different groups of people.
>
> On Wed, Jul 31, 2024 at 11:50 AM Andrew Schofield <
> andrew_schofi...@live.com>
> wrote:
>
> > Hi Josep,
> > I think it’s high time that this was tackled. I suggest that it would be
> > best
> > handled as a KIP because then we have a document which can be discussed
> > and improved, followed by a formal vote.
> >
> > A standard set of terms with agreed meanings would be very helpful for
> > some of the larger KIPs which take many releases to be properly ready for
> > prime time. Most KIPs don’t need this, but a handful definitely do.
> >
> > Personally, I like the sequence that KIP-848 has taken, moving from Early
> > Access, to Preview, and finally complete. I intend to follow the same
> > sequence
> > for KIP-932.
> >
> > Thanks,
> > Andrew
> >
> > > On 31 Jul 2024, at 10:15, Josep Prat 
> > wrote:
> > >
> > > Also as part of this discussion I would like to flag that we need to be
> > > able to know how we can flag this properly so it's known for the
> Release
> > > Manager.
> > > For example, a KIP is approved, the Jira associated with it is being
> > worked
> > > on. Release happens, Jira is still open, how can we flag that this KIP
> is
> > > in early access, or preview?
> > >
> > > Best,
> > >
> > > On Wed, Jul 31, 2024 at 11:03 AM Josep Prat 
> wrote:
> > >
> > >> Hi Kafka devs,
> > >>
> > >> Lately we started using "early access", "production ready" and also
> > >> "preview" to determine the grade of "production readiness" of the
> > features
> > >> we deliver to our community.
> > >> However, as far as I know, there is no official definition from the
> > Apache
> > >> Kafka side on which are the graduation steps for features and what
> type
> > of
> > >> "guarantees" each of these offer.
> > >>
> > >> I think we should agree on which terms we should use and what each of
> > >> these exactly mean in terms of reliability. So far it seems we have
> this
> > >> graduation steps:
> > >> - Early Access: Feature is just complete but not yet fully polished
> and
> > >> maybe not used in production in many environments
> > >> - Preview: Feature was early access before and it underwent at least a
> > >> cycle of improvements and fixes and it's used in some production
> > >> environments maybe
> > >> - Production ready: Feature is officially released and it fulfills the
> > >> expected initial needs
> > >>
> > >> Note that we don't offer any guarantees or SLA/SLO in the classical
> > term.
> > >>
> > >> Is this something we can agree on? What do those terms mean to you? Do
> > we
> > >> need more steps? Or do we need less steps?
> > >>
> > >> Best,
> > >> --
> > >> [image: Aiven] 
> > >>
> > >> *Josep Prat*
> > >> Open Source Engineering Director, *Aiven*
> > >> josep.p...@aiven.io   |   +491715557497
> > >> aiven.io    |
> > >> 
> > >>    <
> > https://twitter.com/aiven_io>
> > >> *Aiven Deutschland GmbH*
> > >> Alexanderufer 3-7, 10117 Berlin
> > >> Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen,
> > >> Anna Richardson, Kenneth Chen
> > >> Amtsgericht Charlottenburg, HRB 209739 B
> > >>
> > >
> > >
> > > --
> > > [image: Aiven] 
> > >
> > > *Josep Prat*
> > > Open Source Engineering Director, *Aiven*
> > > josep.p...@aiven.io   |   +491715557497
> > > aiven.io    |   <
> > https://www.facebook.com/aivencloud>
> > >     <
> > https://twitter.com/aiven_io>
> > > *Aiven Deutschland GmbH*
> > > Alexanderufer 3-7, 10117 Berlin
> > > Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen,
> > > Anna Richardson, Kenneth Chen
> > > Amtsgericht Charlottenburg, HRB 209739 B
> >
> >
>
> --
> [image: Aiven] 
>
> *Josep Prat*
> Open Source Engineering Director, *Aiven*
> josep.p...@aiven.io   |   +491715557497
> aiven.io    |    >
>      <
> https://twitter.com/aiven_io>
> *Aiven Deutschland GmbH*
> Alexanderufer 3-7, 10117 Berlin
> Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen,
> Anna Richardson, Kenneth Chen
> Amtsgericht Charlottenburg, HRB 209739 B
>

Re: [DISCUSS] Graduation steps for Features

[Josep Prat]

Hi David,

One of the problems I see is that the KIP index page has a 1-to-many
relationship between KIP and release. I guess we might want to turn this to
a many-to-many qualified relationship. Otherwise it might be complicated
for the release manager or the KIP driver(s) to keep the Release Plan page
up-to-date for the different steps.

Another alternative would be to have special sub-tasks in JIRA that would
indicate the state of the KIP, then using the "fixed version" label they'll
be included in the release notes and the Release Manager can look for these
special ones when writing announcements or making sure the release notes
are up-to-date.

Best,

On Wed, Jul 31, 2024 at 3:54 PM David Jacot 
wrote:

> Hi Josep,
>
> Thanks for starting the discussion.
>
> We used Early Access, Preview and GA (or Production Ready) for KIP-848 and
> I find it pretty nice. We could add the tentative release plan to the KIP's
> header and it could be used as the source of truth.
>
> Best,
> David
>
> On Wed, Jul 31, 2024 at 11:53 AM Josep Prat 
> wrote:
>
> > Hi Andrew,
> > I can definitely write a KIP, but before doing so I'd like to gather some
> > feedback from the community around these steps and how they are perceived
> > by different groups of people.
> >
> > On Wed, Jul 31, 2024 at 11:50 AM Andrew Schofield <
> > andrew_schofi...@live.com>
> > wrote:
> >
> > > Hi Josep,
> > > I think it’s high time that this was tackled. I suggest that it would
> be
> > > best
> > > handled as a KIP because then we have a document which can be discussed
> > > and improved, followed by a formal vote.
> > >
> > > A standard set of terms with agreed meanings would be very helpful for
> > > some of the larger KIPs which take many releases to be properly ready
> for
> > > prime time. Most KIPs don’t need this, but a handful definitely do.
> > >
> > > Personally, I like the sequence that KIP-848 has taken, moving from
> Early
> > > Access, to Preview, and finally complete. I intend to follow the same
> > > sequence
> > > for KIP-932.
> > >
> > > Thanks,
> > > Andrew
> > >
> > > > On 31 Jul 2024, at 10:15, Josep Prat 
> > > wrote:
> > > >
> > > > Also as part of this discussion I would like to flag that we need to
> be
> > > > able to know how we can flag this properly so it's known for the
> > Release
> > > > Manager.
> > > > For example, a KIP is approved, the Jira associated with it is being
> > > worked
> > > > on. Release happens, Jira is still open, how can we flag that this
> KIP
> > is
> > > > in early access, or preview?
> > > >
> > > > Best,
> > > >
> > > > On Wed, Jul 31, 2024 at 11:03 AM Josep Prat 
> > wrote:
> > > >
> > > >> Hi Kafka devs,
> > > >>
> > > >> Lately we started using "early access", "production ready" and also
> > > >> "preview" to determine the grade of "production readiness" of the
> > > features
> > > >> we deliver to our community.
> > > >> However, as far as I know, there is no official definition from the
> > > Apache
> > > >> Kafka side on which are the graduation steps for features and what
> > type
> > > of
> > > >> "guarantees" each of these offer.
> > > >>
> > > >> I think we should agree on which terms we should use and what each
> of
> > > >> these exactly mean in terms of reliability. So far it seems we have
> > this
> > > >> graduation steps:
> > > >> - Early Access: Feature is just complete but not yet fully polished
> > and
> > > >> maybe not used in production in many environments
> > > >> - Preview: Feature was early access before and it underwent at
> least a
> > > >> cycle of improvements and fixes and it's used in some production
> > > >> environments maybe
> > > >> - Production ready: Feature is officially released and it fulfills
> the
> > > >> expected initial needs
> > > >>
> > > >> Note that we don't offer any guarantees or SLA/SLO in the classical
> > > term.
> > > >>
> > > >> Is this something we can agree on? What do those terms mean to you?
> Do
> > > we
> > > >> need more steps? Or do we need less steps?
> > > >>
> > > >> Best,
> > > >> --
> > > >> [image: Aiven] 
> > > >>
> > > >> *Josep Prat*
> > > >> Open Source Engineering Director, *Aiven*
> > > >> josep.p...@aiven.io   |   +491715557497
> > > >> aiven.io    |
> > > >> 
> > > >>    <
> > > https://twitter.com/aiven_io>
> > > >> *Aiven Deutschland GmbH*
> > > >> Alexanderufer 3-7, 10117 Berlin
> > > >> Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen,
> > > >> Anna Richardson, Kenneth Chen
> > > >> Amtsgericht Charlottenburg, HRB 209739 B
> > > >>
> > > >
> > > >
> > > > --
> > > > [image: Aiven] 
> > > >
> > > > *Josep Prat*
> > > > Open Source Engineering Director, *Aiven*
> > > > josep.p...@aiven.io   |   +491715557497
> > > > aiven.io    |   <
> > > https://www.facebook.com/aivencloud>
> > > >     <
> > > https:

Re: [DISCUSS] Graduation steps for Features

[David Jacot]

We could also use a hierarchie: KIP Parent Jira > Milestones Jiras > Tasks.

Best,
David

On Wed, Jul 31, 2024 at 4:22 PM Josep Prat 
wrote:

> Hi David,
>
> One of the problems I see is that the KIP index page has a 1-to-many
> relationship between KIP and release. I guess we might want to turn this to
> a many-to-many qualified relationship. Otherwise it might be complicated
> for the release manager or the KIP driver(s) to keep the Release Plan page
> up-to-date for the different steps.
>
> Another alternative would be to have special sub-tasks in JIRA that would
> indicate the state of the KIP, then using the "fixed version" label they'll
> be included in the release notes and the Release Manager can look for these
> special ones when writing announcements or making sure the release notes
> are up-to-date.
>
> Best,
>
> On Wed, Jul 31, 2024 at 3:54 PM David Jacot 
> wrote:
>
> > Hi Josep,
> >
> > Thanks for starting the discussion.
> >
> > We used Early Access, Preview and GA (or Production Ready) for KIP-848
> and
> > I find it pretty nice. We could add the tentative release plan to the
> KIP's
> > header and it could be used as the source of truth.
> >
> > Best,
> > David
> >
> > On Wed, Jul 31, 2024 at 11:53 AM Josep Prat  >
> > wrote:
> >
> > > Hi Andrew,
> > > I can definitely write a KIP, but before doing so I'd like to gather
> some
> > > feedback from the community around these steps and how they are
> perceived
> > > by different groups of people.
> > >
> > > On Wed, Jul 31, 2024 at 11:50 AM Andrew Schofield <
> > > andrew_schofi...@live.com>
> > > wrote:
> > >
> > > > Hi Josep,
> > > > I think it’s high time that this was tackled. I suggest that it would
> > be
> > > > best
> > > > handled as a KIP because then we have a document which can be
> discussed
> > > > and improved, followed by a formal vote.
> > > >
> > > > A standard set of terms with agreed meanings would be very helpful
> for
> > > > some of the larger KIPs which take many releases to be properly ready
> > for
> > > > prime time. Most KIPs don’t need this, but a handful definitely do.
> > > >
> > > > Personally, I like the sequence that KIP-848 has taken, moving from
> > Early
> > > > Access, to Preview, and finally complete. I intend to follow the same
> > > > sequence
> > > > for KIP-932.
> > > >
> > > > Thanks,
> > > > Andrew
> > > >
> > > > > On 31 Jul 2024, at 10:15, Josep Prat 
> > > > wrote:
> > > > >
> > > > > Also as part of this discussion I would like to flag that we need
> to
> > be
> > > > > able to know how we can flag this properly so it's known for the
> > > Release
> > > > > Manager.
> > > > > For example, a KIP is approved, the Jira associated with it is
> being
> > > > worked
> > > > > on. Release happens, Jira is still open, how can we flag that this
> > KIP
> > > is
> > > > > in early access, or preview?
> > > > >
> > > > > Best,
> > > > >
> > > > > On Wed, Jul 31, 2024 at 11:03 AM Josep Prat 
> > > wrote:
> > > > >
> > > > >> Hi Kafka devs,
> > > > >>
> > > > >> Lately we started using "early access", "production ready" and
> also
> > > > >> "preview" to determine the grade of "production readiness" of the
> > > > features
> > > > >> we deliver to our community.
> > > > >> However, as far as I know, there is no official definition from
> the
> > > > Apache
> > > > >> Kafka side on which are the graduation steps for features and what
> > > type
> > > > of
> > > > >> "guarantees" each of these offer.
> > > > >>
> > > > >> I think we should agree on which terms we should use and what each
> > of
> > > > >> these exactly mean in terms of reliability. So far it seems we
> have
> > > this
> > > > >> graduation steps:
> > > > >> - Early Access: Feature is just complete but not yet fully
> polished
> > > and
> > > > >> maybe not used in production in many environments
> > > > >> - Preview: Feature was early access before and it underwent at
> > least a
> > > > >> cycle of improvements and fixes and it's used in some production
> > > > >> environments maybe
> > > > >> - Production ready: Feature is officially released and it fulfills
> > the
> > > > >> expected initial needs
> > > > >>
> > > > >> Note that we don't offer any guarantees or SLA/SLO in the
> classical
> > > > term.
> > > > >>
> > > > >> Is this something we can agree on? What do those terms mean to
> you?
> > Do
> > > > we
> > > > >> need more steps? Or do we need less steps?
> > > > >>
> > > > >> Best,
> > > > >> --
> > > > >> [image: Aiven] 
> > > > >>
> > > > >> *Josep Prat*
> > > > >> Open Source Engineering Director, *Aiven*
> > > > >> josep.p...@aiven.io   |   +491715557497
> > > > >> aiven.io    |
> > > > >> 
> > > > >>    <
> > > > https://twitter.com/aiven_io>
> > > > >> *Aiven Deutschland GmbH*
> > > > >> Alexanderufer 3-7, 10117 Berlin
> > > > >> Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen,
> > > > >> Anna Richard

Re: [DISCUSS] KIP-1016 Make MM2 heartbeats topic name configurable

[Kondrát Bertalan]

Hi All,

*Viktor *
I was wrong sorry, Chris is right we have the per replication flow
configuration for free.

*Chris, regarding your comments.*
*>IMO we don't need the "default." prefix*
Sure I can do that, my intention with that was to make it clear for the
users, that this property is not compatible with anything else but the
DefaultReplicationPolicy

*>I'm also a little hazy on the motivation for the change. Just out of*



*curiosity, what exactly is meant by "the "heartbeats" topics of
othersystems" in the Jira ticket's description? Are we trying to
betteraccommodate cases where other harder-to-configure systems (like a
pickysource connector, for example) *
Exactly, we had a customer with a MongoDb connector if I recall it
correctly, and for some reason they could not change the topic name on
connector side.

*Omania*
You are right, I did not notice that my solution is in the rejected
alternatives of KIP-690. I can't argue with your point, and I understand
that the configuration of the mirror maker is not straight forward with all
that options. But those properties are hidden from the user with a
reasonable default value, so in my opinion it should not confuse the users,
but it is there if someone want to modify it.
Renaming an internal topic should not require implementing a new
replication policy class and attaching it to the classpath in my opinion.
I let you and the community decide whether is it worth to reconsider the
previous solution. (I have updated the KIP)

Thanks
Berci

On Fri, Feb 2, 2024 at 4:31 PM Omnia Ibrahim 
wrote:

> I noticed you have something addressed in the rejected alternatives in
> your KIP regarding KIP-690, however, as the KIP is implementing one of the
> previous rejected alternatives for KIP-690 I think it would be nice to
> update the motivation section in the new KIP with some clarification for
> why what was mentioned in the rejected alternatives in KIP-690 isn’t valid
> anymore.
>
> > On 2 Feb 2024, at 15:23, Omnia Ibrahim  wrote:
> >
> > Hi
> > Thanks for the KIP. I don’t know if you had a look before into the
> rejection alternatives in KIP-690
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-690%3A+Add+additional+configuration+to+control+MirrorMaker+2+internal+topics+naming+convention#KIP690:AddadditionalconfigurationtocontrolMirrorMaker2internaltopicsnamingconvention-RejectedAlternatives
> or not but one of the things we were worried about is the amount of
> configuration MM2 already has at the moment specially for advanced and
> complicated replication between multiple clusters. This why we bin-backing
> on the existing of replication policy configs to allow users to define
> their own replication policy with their customised internal topics.
> >
> > Is the assumption now different meaning that we are okay to add more
> configs for mm2? If so can you mention this in the KIP as it was rejected
> before in another KIP
> >
> > Also wouldn’t this mean that we now have two different ways to set the
> `heartbeat` topic name one by this config and another one by overriding the
> replication policy class? It maybe worth mentioning in your KIP why you are
> adding this other path to configure heartbeat topic with single config
> instead of the existing route which providing replication policy with the
> new names of internal topics. And when users should opt-in for each one of
> them.
> >
> > Thanks
> >
> >> On 22 Jan 2024, at 19:52, Chris Egerton 
> wrote:
> >>
> >> Hi Berci,
> >>
> >> Thanks for the KIP!
> >>
> >> IMO we don't need the "default." prefix for the new property, and it
> >> deviates a bit from the precedent set by properties like
> >> "replication.policy.internal.topic.separator.enabled". I think we can
> just
> >> call it "replication.policy.heartbeats.topic", or if we really want to
> be
> >> precise, "replication.policy.heartbeats.topic.name".
> >>
> >> Regarding multiple source->target pairs, won't we get support for this
> for
> >> free if we add the new property to the DefaultReplicationPolicy class?
> IIRC
> >> it's already possible to configure replication policies on a
> >> per-replication-flow basis with that syntax, I don't see why this
> wouldn't
> >> be the case for the new property.
> >>
> >> I'm also a little hazy on the motivation for the change. Just out of
> >> curiosity, what exactly is meant by "the "heartbeats" topics of other
> >> systems" in the Jira ticket's description? Are we trying to better
> >> accommodate cases where other harder-to-configure systems (like a picky
> >> source connector, for example) create and use a "heartbeats" topic, or
> are
> >> we trying to enable multiple MM2 heartbeat connectors to target the same
> >> Kafka cluster? I can understand the former as a niche but possible
> scenario
> >> and one that we can make room for, but the latter is a bit harder to
> >> justify short of, e.g., fine-tuning the heartbeat emission interval
> based
> >> on the eventual target of the replication flow that will be 

[jira] [Created] (KAFKA-17229) Multiple punctuators that together exceed the transaction timeout cause ProducerFencedException

[Trevan Richins (Jira)]

Trevan Richins created KAFKA-17229:
--

 Summary: Multiple punctuators that together exceed the transaction 
timeout cause ProducerFencedException
 Key: KAFKA-17229
 URL: https://issues.apache.org/jira/browse/KAFKA-17229
 Project: Kafka
  Issue Type: Bug
  Components: streams
Affects Versions: 3.8.0
Reporter: Trevan Richins
 Attachments: always-forward-failure.log, topic-input-failure.log

If a single StreamThread has multiple punctuators tasks and the sum total of 
them exceeds the transaction timeout setting, ProducerFencedExceptions will 
occur.

For example, in my test case, I have a input topic with 10 partitions, a 
processor with a punctuator that just sleeps for 5 seconds (the transaction 
timeout is 10s so it finishes within the timeout), and an output topic.  The 
punctuators run every 30 seconds (wall clock).  Once the app is running and is 
inside one of the punctuators, I put one record in the input topic.  The 
punctuators will all finish and the record will be seen and read but it won't 
commit because the punctuators run again (since it has been 30s since they last 
started).  After the punctuators finish this second time, it will try to commit 
the transaction that it started 50 seconds ago and will trigger the 
ProducerFencedException.

Another test case, with the same scenario, is having the punctuators forward 
something.  This also causes a ProducerFencedException because the first 
punctuator starts a transaction but it doesn't commit the transaction till all 
of the punctuators are done and that is long after the transaction timeout.

The issue doesn't exist if there is only one partition as the single punctuator 
will finish within the transaction timeout.  It is only whene there are 
multiple punctuators that exceed the transaction timeout in total.

It feels like what is needed is for kafka to check after each punctuator if 
there is data that needs to be committed.  If there is, it commits then.

 

I've attached a log of the first test case.  It is called 
"topic-input-failure.log".  It starts after the punctuators run the first time. 
 It shows the record being received and the transaction starting.  Then it runs 
the punctuators again and they each sleep for 5 seconds.  Once they are done, 
it triggers a ProducerFencedException.

I've attached a log for the second test case.  It is called 
"always-forward-failure.log".  It starts when the punctuators run the first 
time.  It shows the punctuators forwarding a record and sleeping for 5 seconds. 
 In this case, only 5 punctuators run as a group.  An 
InvalidProducerEpochException occurs after the 5th punctuator finishes.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] Graduation steps for Features

[Josep Prat]

That sounds like a good idea. It will definitely clarify upfront what it
means "Early Access", "Preview" and "Production Ready" for each KIP.
I was looking at JIra and it seems we only have a 2 level hierarchy:
Task(different types of) and Sub-Task. So we would need to change this. I
was also thinking if we could use epics for this, but Epics seem to be only
able to contain tasks and not sub-tasks.

Obviously this would only make sense for big KIPs.

Another thing that we would need is how the release manager can easily see
which KIPs change graduation step in the given release. The easy way would
be use the milestone Jiras (if we can have them) containing the KIP number
in the title to be easily visible.

Best,

On Wed, Jul 31, 2024 at 5:08 PM David Jacot 
wrote:

> We could also use a hierarchie: KIP Parent Jira > Milestones Jiras > Tasks.
>
> Best,
> David
>
> On Wed, Jul 31, 2024 at 4:22 PM Josep Prat 
> wrote:
>
> > Hi David,
> >
> > One of the problems I see is that the KIP index page has a 1-to-many
> > relationship between KIP and release. I guess we might want to turn this
> to
> > a many-to-many qualified relationship. Otherwise it might be complicated
> > for the release manager or the KIP driver(s) to keep the Release Plan
> page
> > up-to-date for the different steps.
> >
> > Another alternative would be to have special sub-tasks in JIRA that would
> > indicate the state of the KIP, then using the "fixed version" label
> they'll
> > be included in the release notes and the Release Manager can look for
> these
> > special ones when writing announcements or making sure the release notes
> > are up-to-date.
> >
> > Best,
> >
> > On Wed, Jul 31, 2024 at 3:54 PM David Jacot  >
> > wrote:
> >
> > > Hi Josep,
> > >
> > > Thanks for starting the discussion.
> > >
> > > We used Early Access, Preview and GA (or Production Ready) for KIP-848
> > and
> > > I find it pretty nice. We could add the tentative release plan to the
> > KIP's
> > > header and it could be used as the source of truth.
> > >
> > > Best,
> > > David
> > >
> > > On Wed, Jul 31, 2024 at 11:53 AM Josep Prat
>  > >
> > > wrote:
> > >
> > > > Hi Andrew,
> > > > I can definitely write a KIP, but before doing so I'd like to gather
> > some
> > > > feedback from the community around these steps and how they are
> > perceived
> > > > by different groups of people.
> > > >
> > > > On Wed, Jul 31, 2024 at 11:50 AM Andrew Schofield <
> > > > andrew_schofi...@live.com>
> > > > wrote:
> > > >
> > > > > Hi Josep,
> > > > > I think it’s high time that this was tackled. I suggest that it
> would
> > > be
> > > > > best
> > > > > handled as a KIP because then we have a document which can be
> > discussed
> > > > > and improved, followed by a formal vote.
> > > > >
> > > > > A standard set of terms with agreed meanings would be very helpful
> > for
> > > > > some of the larger KIPs which take many releases to be properly
> ready
> > > for
> > > > > prime time. Most KIPs don’t need this, but a handful definitely do.
> > > > >
> > > > > Personally, I like the sequence that KIP-848 has taken, moving from
> > > Early
> > > > > Access, to Preview, and finally complete. I intend to follow the
> same
> > > > > sequence
> > > > > for KIP-932.
> > > > >
> > > > > Thanks,
> > > > > Andrew
> > > > >
> > > > > > On 31 Jul 2024, at 10:15, Josep Prat  >
> > > > > wrote:
> > > > > >
> > > > > > Also as part of this discussion I would like to flag that we need
> > to
> > > be
> > > > > > able to know how we can flag this properly so it's known for the
> > > > Release
> > > > > > Manager.
> > > > > > For example, a KIP is approved, the Jira associated with it is
> > being
> > > > > worked
> > > > > > on. Release happens, Jira is still open, how can we flag that
> this
> > > KIP
> > > > is
> > > > > > in early access, or preview?
> > > > > >
> > > > > > Best,
> > > > > >
> > > > > > On Wed, Jul 31, 2024 at 11:03 AM Josep Prat  >
> > > > wrote:
> > > > > >
> > > > > >> Hi Kafka devs,
> > > > > >>
> > > > > >> Lately we started using "early access", "production ready" and
> > also
> > > > > >> "preview" to determine the grade of "production readiness" of
> the
> > > > > features
> > > > > >> we deliver to our community.
> > > > > >> However, as far as I know, there is no official definition from
> > the
> > > > > Apache
> > > > > >> Kafka side on which are the graduation steps for features and
> what
> > > > type
> > > > > of
> > > > > >> "guarantees" each of these offer.
> > > > > >>
> > > > > >> I think we should agree on which terms we should use and what
> each
> > > of
> > > > > >> these exactly mean in terms of reliability. So far it seems we
> > have
> > > > this
> > > > > >> graduation steps:
> > > > > >> - Early Access: Feature is just complete but not yet fully
> > polished
> > > > and
> > > > > >> maybe not used in production in many environments
> > > > > >> - Preview: Feature was early access before and it underwent at
> > > least a
> > > > > >> cycle of impr

Re: [DISCUSS] Graduation steps for Features

[Colin McCabe]

Hi Josep,

Thanks for bringing up this discussion.

My impression was that "preview" was earlier than early access.

In my mind, "preview" somewhat implies that the feature could be incomplete or 
partial. For example, if someone said their game was a "preview" I would expect 
kind of a demo.

Early access (EA) means people can "access" it but it's "early". In other 
words, use at your own risk.

What you are calling "production ready," I think we've sometimes called 
generally available (GA) in the past. I don't have a strong preference on terms.

I think no matter how much standardization we do, there will always be a lot of 
discretion in these terms :) But maybe this helps clarify how they've been used 
in the past?

best,
Colin


On Wed, Jul 31, 2024, at 02:03, Josep Prat wrote:
> Hi Kafka devs,
>
> Lately we started using "early access", "production ready" and also
> "preview" to determine the grade of "production readiness" of the features
> we deliver to our community.
> However, as far as I know, there is no official definition from the Apache
> Kafka side on which are the graduation steps for features and what type of
> "guarantees" each of these offer.
>
> I think we should agree on which terms we should use and what each of these
> exactly mean in terms of reliability. So far it seems we have this
> graduation steps:
> - Early Access: Feature is just complete but not yet fully polished and
> maybe not used in production in many environments
> - Preview: Feature was early access before and it underwent at least a
> cycle of improvements and fixes and it's used in some production
> environments maybe
> - Production ready: Feature is officially released and it fulfills the
> expected initial needs
>
> Note that we don't offer any guarantees or SLA/SLO in the classical term.
>
> Is this something we can agree on? What do those terms mean to you? Do we
> need more steps? Or do we need less steps?
>
> Best,
> -- 
> [image: Aiven] 
>
> *Josep Prat*
> Open Source Engineering Director, *Aiven*
> josep.p...@aiven.io   |   +491715557497
> aiven.io    |   
>      
> *Aiven Deutschland GmbH*
> Alexanderufer 3-7, 10117 Berlin
> Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen,
> Anna Richardson, Kenneth Chen
> Amtsgericht Charlottenburg, HRB 209739 B

Re: [DISCUSS] Graduation steps for Features

[Josep Prat]

Hi Colin,

I'm not attached to any names, I'm happy to change them.
But from what I can guess from KIP-848 the order is EA -> Preview -> GA
Maybe I'm mistaken.

This is why this discussion is important so we all agree on the number of
steps, order and names.

I know finding a universal solution is maybe impossible but if we can find
one that fits most of the feature releases we had and we are planning
currently would be great.

Best,

--
Josep Prat
Open Source Engineering Director, Aiven
josep.p...@aiven.io   |   +491715557497 | aiven.io
Aiven Deutschland GmbH
Alexanderufer 3-7, 10117 Berlin
Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen,
Anna Richardson, Kenneth Chen
Amtsgericht Charlottenburg, HRB 209739 B

On Wed, Jul 31, 2024, 18:31 Colin McCabe  wrote:

> Hi Josep,
>
> Thanks for bringing up this discussion.
>
> My impression was that "preview" was earlier than early access.
>
> In my mind, "preview" somewhat implies that the feature could be
> incomplete or partial. For example, if someone said their game was a
> "preview" I would expect kind of a demo.
>
> Early access (EA) means people can "access" it but it's "early". In other
> words, use at your own risk.
>
> What you are calling "production ready," I think we've sometimes called
> generally available (GA) in the past. I don't have a strong preference on
> terms.
>
> I think no matter how much standardization we do, there will always be a
> lot of discretion in these terms :) But maybe this helps clarify how
> they've been used in the past?
>
> best,
> Colin
>
>
> On Wed, Jul 31, 2024, at 02:03, Josep Prat wrote:
> > Hi Kafka devs,
> >
> > Lately we started using "early access", "production ready" and also
> > "preview" to determine the grade of "production readiness" of the
> features
> > we deliver to our community.
> > However, as far as I know, there is no official definition from the
> Apache
> > Kafka side on which are the graduation steps for features and what type
> of
> > "guarantees" each of these offer.
> >
> > I think we should agree on which terms we should use and what each of
> these
> > exactly mean in terms of reliability. So far it seems we have this
> > graduation steps:
> > - Early Access: Feature is just complete but not yet fully polished and
> > maybe not used in production in many environments
> > - Preview: Feature was early access before and it underwent at least a
> > cycle of improvements and fixes and it's used in some production
> > environments maybe
> > - Production ready: Feature is officially released and it fulfills the
> > expected initial needs
> >
> > Note that we don't offer any guarantees or SLA/SLO in the classical term.
> >
> > Is this something we can agree on? What do those terms mean to you? Do we
> > need more steps? Or do we need less steps?
> >
> > Best,
> > --
> > [image: Aiven] 
> >
> > *Josep Prat*
> > Open Source Engineering Director, *Aiven*
> > josep.p...@aiven.io   |   +491715557497
> > aiven.io    |   <
> https://www.facebook.com/aivencloud>
> >      <
> https://twitter.com/aiven_io>
> > *Aiven Deutschland GmbH*
> > Alexanderufer 3-7, 10117 Berlin
> > Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen,
> > Anna Richardson, Kenneth Chen
> > Amtsgericht Charlottenburg, HRB 209739 B
>

Re: [VOTE] KIP-877: Mechanism for plugins and connectors to register metrics

[Mickael Maison]

Hi,

Bumping this thread to get some more votes and/or feedback.

As I restarted the vote on June 10 after major changes, I'm only
counting votes since then.
So we have 1 binding (Chris) and 1 non-binding (Hector) votes.

Thanks,
Mickael

On Mon, Jul 8, 2024 at 8:40 PM Hector Geraldino (BLOOMBERG/ 919 3RD A)
 wrote:
>
> This will help eliminate some boilerplate code we have for our connectors.
>
> +1 (non-binding)
>
> From: dev@kafka.apache.org At: 06/25/24 04:30:27 UTC-4:00To:  
> dev@kafka.apache.org
> Subject: Re: [VOTE] KIP-877: Mechanism for plugins and connectors to register 
> metrics
>
> Bumping this thread.
>
> Let me know if you have any feedback.
>
> Thanks,
> Mickael
>
> On Mon, Jun 10, 2024 at 1:44 PM Chris Egerton  wrote:
> >
> > +1 (binding), thanks Mickael!
> >
> > On Mon, Jun 10, 2024, 04:24 Mickael Maison  wrote:
> >
> > > Hi,
> > >
> > > Following the feedback in the DISCUSS thread, I made significant
> > > changes to the proposal. So I'd like to restart a vote for KIP-877:
> > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-877%3A+Mechanism+for+plugi
> ns+and+connectors+to+register+metrics
> > >
> > > Thanks,
> > > Mickael
> > >
> > > On Thu, Jan 25, 2024 at 2:59 AM Tom Bentley  wrote:
> > > >
> > > > Hi Mickael,
> > > >
> > > > You'll have seen that I left some comments on the discussion thread, but
> > > > they're minor enough that I'm happy to vote +1 here.
> > > >
> > > > Thanks,
> > > >
> > > > Tom
> > > >
> > > > On Thu, 11 Jan 2024 at 06:14, Mickael Maison 
> > > > wrote:
> > > >
> > > > > Bumping this thread since I've not seen any feedback.
> > > > >
> > > > > Thanks,
> > > > > Mickael
> > > > >
> > > > > On Tue, Dec 19, 2023 at 10:03 AM Mickael Maison
> > > > >  wrote:
> > > > > >
> > > > > > Hi,
> > > > > >
> > > > > > I'd like to start a vote on KIP-877:
> > > > > >
> > > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-877%3A+Mechanism+for+plugi
> ns+and+connectors+to+register+metrics
> > > > > >
> > > > > > Let me know if you have any feedback.
> > > > > >
> > > > > > Thanks,
> > > > > > Mickael
> > > > >
> > > > >
> > >
>
>

[jira] [Created] (KAFKA-17230) Kafka consumer client doesn't report node request-latency metrics

[Apoorv Mittal (Jira)]

Apoorv Mittal created KAFKA-17230:
-

 Summary: Kafka consumer client doesn't report node request-latency 
metrics
 Key: KAFKA-17230
 URL: https://issues.apache.org/jira/browse/KAFKA-17230
 Project: Kafka
  Issue Type: Bug
  Components: clients
Reporter: Apoorv Mittal
Assignee: Apoorv Mittal


Kafka Consumer client registers node/connection latency metrics in 
[Selector.java|https://github.com/apache/kafka/blob/9d634629f29cd402a723d7e7bece18c8099065a4/clients/src/main/java/org/apache/kafka/common/network/Selector.java#L1359]
 but the values against the metric is never recorded. This seems to be an issue 
since inception. However, the same metric is also created for Kafka Producer 
but the value is recorded for Kafka Producer in 
[Sender.java|https://github.com/apache/kafka/blob/9d634629f29cd402a723d7e7bece18c8099065a4/clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java#L1092].
 Hence node - request-latency-max and request-latency-avg appears correctly for 
Kafka Producer but is NaN for Kafka Consumer.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] Graduation steps for Features

[Kirk True]

Hi Josep,

Thanks for starting this conversation!

Yes, for KIPs that span multiple releases, clearly defining the KIP’s state for 
each release is very important. Perhaps the KIP template could be updated to 
include a “Release Plan” section that includes entries that include:

- Version: future Apache Kafka release
  - Status: one of Preview, Early access, Production, etc.
  - Jira: umbrella Jira under which other Jiras relevant to that release are 
kept
  - Notes: Free-form notes as needed by KIP author

Using KIP-848 as an example, the “Release Plan” section could look like this:

- 3.7
  - Status: Early access
  - Jira: KAFKA-123
  - Notes: includes support for user-assigned partitions but not subscriptions, 
server-side regex not included
- 3.8
  - Status: Preview
  - Jira: KAFKA-234
  - Notes: includes support for subscriptions and lots of bug fixes
- 4.0
  - Status: Production
  - Jira: KAFKA-345
  - Notes: includes feature parity with “CLASSIC” consumer groups

Of course, the contents of the “Release Plan” section may need to change over 
time, and it’s important that the KIP author(s) update it accordingly.

Just a thought.

Thanks,
Kirk

> On Jul 31, 2024, at 9:43 AM, Josep Prat  wrote:
> 
> Hi Colin,
> 
> I'm not attached to any names, I'm happy to change them.
> But from what I can guess from KIP-848 the order is EA -> Preview -> GA
> Maybe I'm mistaken.
> 
> This is why this discussion is important so we all agree on the number of
> steps, order and names.
> 
> I know finding a universal solution is maybe impossible but if we can find
> one that fits most of the feature releases we had and we are planning
> currently would be great.
> 
> Best,
> 
> --
> Josep Prat
> Open Source Engineering Director, Aiven
> josep.p...@aiven.io   |   +491715557497 | aiven.io
> Aiven Deutschland GmbH
> Alexanderufer 3-7, 10117 Berlin
> Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen,
> Anna Richardson, Kenneth Chen
> Amtsgericht Charlottenburg, HRB 209739 B
> 
> On Wed, Jul 31, 2024, 18:31 Colin McCabe  wrote:
> 
>> Hi Josep,
>> 
>> Thanks for bringing up this discussion.
>> 
>> My impression was that "preview" was earlier than early access.
>> 
>> In my mind, "preview" somewhat implies that the feature could be
>> incomplete or partial. For example, if someone said their game was a
>> "preview" I would expect kind of a demo.
>> 
>> Early access (EA) means people can "access" it but it's "early". In other
>> words, use at your own risk.
>> 
>> What you are calling "production ready," I think we've sometimes called
>> generally available (GA) in the past. I don't have a strong preference on
>> terms.
>> 
>> I think no matter how much standardization we do, there will always be a
>> lot of discretion in these terms :) But maybe this helps clarify how
>> they've been used in the past?
>> 
>> best,
>> Colin
>> 
>> 
>> On Wed, Jul 31, 2024, at 02:03, Josep Prat wrote:
>>> Hi Kafka devs,
>>> 
>>> Lately we started using "early access", "production ready" and also
>>> "preview" to determine the grade of "production readiness" of the
>> features
>>> we deliver to our community.
>>> However, as far as I know, there is no official definition from the
>> Apache
>>> Kafka side on which are the graduation steps for features and what type
>> of
>>> "guarantees" each of these offer.
>>> 
>>> I think we should agree on which terms we should use and what each of
>> these
>>> exactly mean in terms of reliability. So far it seems we have this
>>> graduation steps:
>>> - Early Access: Feature is just complete but not yet fully polished and
>>> maybe not used in production in many environments
>>> - Preview: Feature was early access before and it underwent at least a
>>> cycle of improvements and fixes and it's used in some production
>>> environments maybe
>>> - Production ready: Feature is officially released and it fulfills the
>>> expected initial needs
>>> 
>>> Note that we don't offer any guarantees or SLA/SLO in the classical term.
>>> 
>>> Is this something we can agree on? What do those terms mean to you? Do we
>>> need more steps? Or do we need less steps?
>>> 
>>> Best,
>>> --
>>> [image: Aiven] 
>>> 
>>> *Josep Prat*
>>> Open Source Engineering Director, *Aiven*
>>> josep.p...@aiven.io   |   +491715557497
>>> aiven.io    |   <
>> https://www.facebook.com/aivencloud>
>>>     <
>> https://twitter.com/aiven_io>
>>> *Aiven Deutschland GmbH*
>>> Alexanderufer 3-7, 10117 Berlin
>>> Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen,
>>> Anna Richardson, Kenneth Chen
>>> Amtsgericht Charlottenburg, HRB 209739 B
>> 

Re: [DISCUSS] Graduation steps for Features

[Chris Egerton]

I'd be hesitant to add too much to the KIP template. We should be careful
to keep the barrier to entry low for new users.

On Wed, Jul 31, 2024, 13:24 Kirk True  wrote:

> Hi Josep,
>
> Thanks for starting this conversation!
>
> Yes, for KIPs that span multiple releases, clearly defining the KIP’s
> state for each release is very important. Perhaps the KIP template could be
> updated to include a “Release Plan” section that includes entries that
> include:
>
> - Version: future Apache Kafka release
>   - Status: one of Preview, Early access, Production, etc.
>   - Jira: umbrella Jira under which other Jiras relevant to that release
> are kept
>   - Notes: Free-form notes as needed by KIP author
>
> Using KIP-848 as an example, the “Release Plan” section could look like
> this:
>
> - 3.7
>   - Status: Early access
>   - Jira: KAFKA-123
>   - Notes: includes support for user-assigned partitions but not
> subscriptions, server-side regex not included
> - 3.8
>   - Status: Preview
>   - Jira: KAFKA-234
>   - Notes: includes support for subscriptions and lots of bug fixes
> - 4.0
>   - Status: Production
>   - Jira: KAFKA-345
>   - Notes: includes feature parity with “CLASSIC” consumer groups
>
> Of course, the contents of the “Release Plan” section may need to change
> over time, and it’s important that the KIP author(s) update it accordingly.
>
> Just a thought.
>
> Thanks,
> Kirk
>
> > On Jul 31, 2024, at 9:43 AM, Josep Prat 
> wrote:
> >
> > Hi Colin,
> >
> > I'm not attached to any names, I'm happy to change them.
> > But from what I can guess from KIP-848 the order is EA -> Preview -> GA
> > Maybe I'm mistaken.
> >
> > This is why this discussion is important so we all agree on the number of
> > steps, order and names.
> >
> > I know finding a universal solution is maybe impossible but if we can
> find
> > one that fits most of the feature releases we had and we are planning
> > currently would be great.
> >
> > Best,
> >
> > --
> > Josep Prat
> > Open Source Engineering Director, Aiven
> > josep.p...@aiven.io   |   +491715557497 | aiven.io
> > Aiven Deutschland GmbH
> > Alexanderufer 3-7, 10117 Berlin
> > Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen,
> > Anna Richardson, Kenneth Chen
> > Amtsgericht Charlottenburg, HRB 209739 B
> >
> > On Wed, Jul 31, 2024, 18:31 Colin McCabe  wrote:
> >
> >> Hi Josep,
> >>
> >> Thanks for bringing up this discussion.
> >>
> >> My impression was that "preview" was earlier than early access.
> >>
> >> In my mind, "preview" somewhat implies that the feature could be
> >> incomplete or partial. For example, if someone said their game was a
> >> "preview" I would expect kind of a demo.
> >>
> >> Early access (EA) means people can "access" it but it's "early". In
> other
> >> words, use at your own risk.
> >>
> >> What you are calling "production ready," I think we've sometimes called
> >> generally available (GA) in the past. I don't have a strong preference
> on
> >> terms.
> >>
> >> I think no matter how much standardization we do, there will always be a
> >> lot of discretion in these terms :) But maybe this helps clarify how
> >> they've been used in the past?
> >>
> >> best,
> >> Colin
> >>
> >>
> >> On Wed, Jul 31, 2024, at 02:03, Josep Prat wrote:
> >>> Hi Kafka devs,
> >>>
> >>> Lately we started using "early access", "production ready" and also
> >>> "preview" to determine the grade of "production readiness" of the
> >> features
> >>> we deliver to our community.
> >>> However, as far as I know, there is no official definition from the
> >> Apache
> >>> Kafka side on which are the graduation steps for features and what type
> >> of
> >>> "guarantees" each of these offer.
> >>>
> >>> I think we should agree on which terms we should use and what each of
> >> these
> >>> exactly mean in terms of reliability. So far it seems we have this
> >>> graduation steps:
> >>> - Early Access: Feature is just complete but not yet fully polished and
> >>> maybe not used in production in many environments
> >>> - Preview: Feature was early access before and it underwent at least a
> >>> cycle of improvements and fixes and it's used in some production
> >>> environments maybe
> >>> - Production ready: Feature is officially released and it fulfills the
> >>> expected initial needs
> >>>
> >>> Note that we don't offer any guarantees or SLA/SLO in the classical
> term.
> >>>
> >>> Is this something we can agree on? What do those terms mean to you? Do
> we
> >>> need more steps? Or do we need less steps?
> >>>
> >>> Best,
> >>> --
> >>> [image: Aiven] 
> >>>
> >>> *Josep Prat*
> >>> Open Source Engineering Di
> rector,
> *Aiven*
> >>> josep.p...@aiven.io   |   +491715557497
> >>> aiven.io    |   <
> >> https://www.facebook.com/aivencloud>
> >>>     <
> >> https://twitter.com/aiven_io>
> >

Re: [DISCUSS] KIP-512: make Record Headers available in onAcknowledgement

[Rich C.]

Hi Lianet,

> LM1: Main advantage would be not having to add a header for calculating
latency

I see your points. I want to share our use case for latency measurements.
We have a pipeline: `MySQL -> Debezium -> KafkaA -> Flink -> KafkaB`. In
KIP-512, I simplified it to measure `Flink Sink -> KafkaB`. However, we
also need to measure end-to-end latency from `MySQL -> KafkaB`. The
timestamp from MySQL will be set in a header, so having headers generally
available in onAcknowledgement will help with this.

> LM2: I share Andrew's feeling because Headers are indeed part of the
ProducerRecord. But then headers will in practice simply contain info
related to the record, so it seems sensible to expect to find headers in
the RecordMetadata as you suggest, ok with me

Am I correct in understanding that you see value in both options and are
fine with either one?

Regards,
Rich


On Tue, Jul 30, 2024 at 3:07 PM Lianet M.  wrote:

> Hello Rich, thanks for resurrecting the KIP, seems to fill a gap indeed.
>
> LM1. Specifically related to motivation#1. ProducerRecord already has a
> timestamp, passed into the RecordMetadata, that represents the creation
> time provided on new ProducerRecord, so couldn't we reuse it to avoid the
> extra complexity of having to "include a timestamp in the header when the
> message is sent" to be able to compute latency properly. The challenge of
> course is that that timestamp may be overwritten (and this is the root
> cause of the gap), but that could be resolved just by keeping the original
> time and making it available.
> RecordMetadata would keep a timestamp (passed from the record creation,
> never mutated), and the "effectiveTimestamp" (the one it currently has,
> updated with the broker result based on configs). Main advantage would be
> not having to add a header for calculating latency. The user simply creates
> the record with a timestamp (known existing concept), and we make that
> value accessible in the RecordMetadata (where it exists already at some
> point, but it's mutated). Thoughts?
>
> LM2. Regardless of the point above, if we think having the headers
> available on the onAcknowledgement would be helpful, I definitely see the
> case for both alternatives (headers in RecordMetadata and as param). I
> share Andrew's feeling because Headers are indeed part of the
> ProducerRecord. But then headers will in practice simply contain info
> related to the record, so it seems sensible to expect to find headers in
> the RecordMetadata as you suggest, ok with me.
>
> Thanks!
>
> Lianet
>
> On Mon, Jul 29, 2024 at 9:41 PM Rich C.  wrote:
>
> > Hi Andrew,
> >
> > Thanks for the feedback. I have updated KIP-512 and addressed AS2, AS3,
> and
> > AS4. For AS1, let's wait for further responses from the community.
> >
> >
> > Regards,
> > Rich
> >
> >
> > On Mon, Jul 29, 2024 at 5:59 AM Andrew Schofield <
> > andrew_schofi...@live.com>
> > wrote:
> >
> > > Hi,
> > > Thanks for adding the detail. It seems quite straightforward to
> > > implement in the producer code.
> > >
> > > AS1: Personally, and of course this is a matter of taste and just one
> > > opinion, I don’t like adding Headers to RecordMetadata. It seems to me
> > > that RecordMetadata is information about the record that’s been
> produced
> > > whereas the Headers are really part of the record itself. So, I prefer
> > the
> > > alternative which overloads ProducerInterceptor.onAcknowledgement.
> > >
> > > AS2: ProducerBatch and FutureRecordMetadata are both internal classes
> > > and do not need to be documented in the KIP.
> > >
> > > AS3: This KIP is adding rather than replcaing the constructor for
> > > RecordMetadata.
> > > You should define the value for the Headers if an existing constructor
> > > without headers is used.
> > >
> > > AS4: You should add a method `Headers headers()` to RecordMetadata.
> > >
> > >
> > > I wonder what other community members think about whether it’s a good
> > > idea to extend RecordMetadata with the headers.
> > >
> > > Thanks,
> > > Andrew
> > >
> > > > On 29 Jul 2024, at 05:36, Rich C.  wrote:
> > > >
> > > > Hi all,
> > > >
> > > > Thank you for the positive feedback. I added proposal changes to
> > KIP-512
> > > > and included a FAQ section to address some concerns.
> > > >
> > > > Hi Andrew, yes, this KIP focuses on
> > > > `ProducerInterceptor.onAcknowledgement`. I added FAQ#3 to explain
> that.
> > > >
> > > > Hi Matthias, for your question about "RecordMetadata being Kafka
> > > metadata" in
> > > > this thread
> > > > <
> > >
> >
> https://lists.apache.org/list?dev@kafka.apache.org:lte=1M:make%20Record%20Headers%20available%20in%20onAcknowledgement
> > > >,
> > > > I added FAQ#2 to explain that. If I have missed any documentation
> > > regarding
> > > > the design of RecordMetadata, please let me know.
> > > >
> > > > Regards,
> > > > Rich
> > > >
> > > >
> > > > On Fri, Jul 26, 2024 at 4:00 PM Andrew Schofield <
> > > andrew_schofi...@live.com>
> > > > wrote:
> > > >
> > > >> 

[jira] [Created] (KAFKA-17231) Sn

[Andrew Schofield (Jira)]

Andrew Schofield created KAFKA-17231:


 Summary: Sn
 Key: KAFKA-17231
 URL: https://issues.apache.org/jira/browse/KAFKA-17231
 Project: Kafka
  Issue Type: Sub-task
Reporter: Andrew Schofield






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Jenkins build is unstable: Kafka » Kafka Branch Builder » 3.8 #77

[Apache Jenkins Server]

See 

[jira] [Created] (KAFKA-17232) MirrorCheckpointConnector does not generate task configs if initial consumer group load times out

[Greg Harris (Jira)]

Greg Harris created KAFKA-17232:
---

 Summary: MirrorCheckpointConnector does not generate task configs 
if initial consumer group load times out
 Key: KAFKA-17232
 URL: https://issues.apache.org/jira/browse/KAFKA-17232
 Project: Kafka
  Issue Type: Bug
  Components: mirrormaker
Affects Versions: 3.9.0
Reporter: Greg Harris


The MirrorCheckpointConnector has two operations that read the source consumer 
groups:
 * loadInitialConsumerGroups
 * refreshConsumerGroups

loadInitialConsumerGroups blocks the start() method of the connector, while 
refreshConsumerGroups is asynchronous and runs periodically while the connector 
is running.

loadInitialConsumerGroups may take a long time to execute, and may exceed the 
configured "admin.timeout.ms" used by the Scheduler. This timeout is logged and 
the start() method returns normally. If this happens, the framework will 
generate task configs immediately after start(), before 
loadInitialConsumerGroups can finish, and will generate an empty set of task 
configs: 
[https://github.com/apache/kafka/blob/e2494e6ffb89f8288ed2aeb9b5596c755210bffd/connect/mirror/src/main/java/org/apache/kafka/connect/mirror/MirrorCheckpointConnector.java#L118-L121].

Later, when loadInitialConsumerGroups completes, it will not request task 
reconfiguration, believing it is the initial load operation.

Later still, when refreshConsumerGroups completes, it will not request task 
reconfiguration, as the set of consumer groups has not changed since the 
initial load: 
[https://github.com/apache/kafka/blob/e2494e6ffb89f8288ed2aeb9b5596c755210bffd/connect/mirror/src/main/java/org/apache/kafka/connect/mirror/MirrorCheckpointConnector.java#L173-L180]
 

This leads to a situation where the MirrorCheckpointConnector believes it has 
converged with nothing to update, but actually has consumer groups that are not 
allocated to tasks.

This happens particularly for large, stable Kafka clusters with many consumer 
groups that are not being actively created or deleted.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] Graduation steps for Features

[Andrew Schofield]

Hi,
I would at least like to see agreement on the various stages. I’m used to
Early Access, then Preview, and finally General Availability, but I notice that
at least one other person on this thread had the order different. Also,
General Availability sounds a bit wrong for an open-source project, so maybe
Production Ready is better.

I’d also like to see some guidelines for what to expect for features in the
various stages. For example, prior to Production Ready, I would not expect
a feature to be turned on by default for a new broker.

The point about barrier to entry is fair, but I think this really only applies 
to
a handful of large KIPs which take several releases to come to fruition.
I’m sure we didn’t know when KRaft began which release would be the first
in which it would be production ready. I’m sure we decided to release some early
code long before that and having a proper name for that stage seems appropriate.

Thanks,
Andrew

> On 31 Jul 2024, at 18:30, Chris Egerton  wrote:
>
> I'd be hesitant to add too much to the KIP template. We should be careful
> to keep the barrier to entry low for new users.
>
> On Wed, Jul 31, 2024, 13:24 Kirk True  wrote:
>
>> Hi Josep,
>>
>> Thanks for starting this conversation!
>>
>> Yes, for KIPs that span multiple releases, clearly defining the KIP’s
>> state for each release is very important. Perhaps the KIP template could be
>> updated to include a “Release Plan” section that includes entries that
>> include:
>>
>> - Version: future Apache Kafka release
>>  - Status: one of Preview, Early access, Production, etc.
>>  - Jira: umbrella Jira under which other Jiras relevant to that release
>> are kept
>>  - Notes: Free-form notes as needed by KIP author
>>
>> Using KIP-848 as an example, the “Release Plan” section could look like
>> this:
>>
>> - 3.7
>>  - Status: Early access
>>  - Jira: KAFKA-123
>>  - Notes: includes support for user-assigned partitions but not
>> subscriptions, server-side regex not included
>> - 3.8
>>  - Status: Preview
>>  - Jira: KAFKA-234
>>  - Notes: includes support for subscriptions and lots of bug fixes
>> - 4.0
>>  - Status: Production
>>  - Jira: KAFKA-345
>>  - Notes: includes feature parity with “CLASSIC” consumer groups
>>
>> Of course, the contents of the “Release Plan” section may need to change
>> over time, and it’s important that the KIP author(s) update it accordingly.
>>
>> Just a thought.
>>
>> Thanks,
>> Kirk
>>
>>> On Jul 31, 2024, at 9:43 AM, Josep Prat 
>> wrote:
>>>
>>> Hi Colin,
>>>
>>> I'm not attached to any names, I'm happy to change them.
>>> But from what I can guess from KIP-848 the order is EA -> Preview -> GA
>>> Maybe I'm mistaken.
>>>
>>> This is why this discussion is important so we all agree on the number of
>>> steps, order and names.
>>>
>>> I know finding a universal solution is maybe impossible but if we can
>> find
>>> one that fits most of the feature releases we had and we are planning
>>> currently would be great.
>>>
>>> Best,
>>>
>>> --
>>> Josep Prat
>>> Open Source Engineering Director, Aiven
>>> josep.p...@aiven.io   |   +491715557497 | aiven.io
>>> Aiven Deutschland GmbH
>>> Alexanderufer 3-7, 10117 Berlin
>>> Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen,
>>> Anna Richardson, Kenneth Chen
>>> Amtsgericht Charlottenburg, HRB 209739 B
>>>
>>> On Wed, Jul 31, 2024, 18:31 Colin McCabe  wrote:
>>>
 Hi Josep,

 Thanks for bringing up this discussion.

 My impression was that "preview" was earlier than early access.

 In my mind, "preview" somewhat implies that the feature could be
 incomplete or partial. For example, if someone said their game was a
 "preview" I would expect kind of a demo.

 Early access (EA) means people can "access" it but it's "early". In
>> other
 words, use at your own risk.

 What you are calling "production ready," I think we've sometimes called
 generally available (GA) in the past. I don't have a strong preference
>> on
 terms.

 I think no matter how much standardization we do, there will always be a
 lot of discretion in these terms :) But maybe this helps clarify how
 they've been used in the past?

 best,
 Colin


 On Wed, Jul 31, 2024, at 02:03, Josep Prat wrote:
> Hi Kafka devs,
>
> Lately we started using "early access", "production ready" and also
> "preview" to determine the grade of "production readiness" of the
 features
> we deliver to our community.
> However, as far as I know, there is no official definition from the
 Apache
> Kafka side on which are the graduation steps for features and what type
 of
> "guarantees" each of these offer.
>
> I think we should agree on which terms we should use and what each of
 these
> exactly mean in terms of reliability. So far it seems we have this
> graduation steps:
> - Early Access: Feature is just complete but not yet fully 

Re: [DISCUSS] Graduation steps for Features

[Josep Prat]

Hi,

I think we can keep the KIP template as is and have the extra process in
the Jira area.

EA, Preview and Production Ready (I also kind of like this better) feel to
me more project management while the KIP document itself is more about the
nature of the changes and describing the end result. Except for some KIPs
that inherently describe a process delivered in stages.

Best,
--
Josep Prat
Open Source Engineering Director, Aiven
josep.p...@aiven.io   |   +491715557497 | aiven.io
Aiven Deutschland GmbH
Alexanderufer 3-7, 10117 Berlin
Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen,
Anna Richardson, Kenneth Chen
Amtsgericht Charlottenburg, HRB 209739 B

On Wed, Jul 31, 2024, 21:12 Andrew Schofield 
wrote:

> Hi,
> I would at least like to see agreement on the various stages. I’m used to
> Early Access, then Preview, and finally General Availability, but I notice
> that
> at least one other person on this thread had the order different. Also,
> General Availability sounds a bit wrong for an open-source project, so
> maybe
> Production Ready is better.
>
> I’d also like to see some guidelines for what to expect for features in the
> various stages. For example, prior to Production Ready, I would not expect
> a feature to be turned on by default for a new broker.
>
> The point about barrier to entry is fair, but I think this really only
> applies to
> a handful of large KIPs which take several releases to come to fruition.
> I’m sure we didn’t know when KRaft began which release would be the first
> in which it would be production ready. I’m sure we decided to release some
> early
> code long before that and having a proper name for that stage seems
> appropriate.
>
> Thanks,
> Andrew
>
> > On 31 Jul 2024, at 18:30, Chris Egerton  wrote:
> >
> > I'd be hesitant to add too much to the KIP template. We should be careful
> > to keep the barrier to entry low for new users.
> >
> > On Wed, Jul 31, 2024, 13:24 Kirk True  wrote:
> >
> >> Hi Josep,
> >>
> >> Thanks for starting this conversation!
> >>
> >> Yes, for KIPs that span multiple releases, clearly defining the KIP’s
> >> state for each release is very important. Perhaps the KIP template
> could be
> >> updated to include a “Release Plan” section that includes entries that
> >> include:
> >>
> >> - Version: future Apache Kafka release
> >>  - Status: one of Preview, Early access, Production, etc.
> >>  - Jira: umbrella Jira under which other Jiras relevant to that release
> >> are kept
> >>  - Notes: Free-form notes as needed by KIP author
> >>
> >> Using KIP-848 as an example, the “Release Plan” section could look like
> >> this:
> >>
> >> - 3.7
> >>  - Status: Early access
> >>  - Jira: KAFKA-123
> >>  - Notes: includes support for user-assigned partitions but not
> >> subscriptions, server-side regex not included
> >> - 3.8
> >>  - Status: Preview
> >>  - Jira: KAFKA-234
> >>  - Notes: includes support for subscriptions and lots of bug fixes
> >> - 4.0
> >>  - Status: Production
> >>  - Jira: KAFKA-345
> >>  - Notes: includes feature parity with “CLASSIC” consumer groups
> >>
> >> Of course, the contents of the “Release Plan” section may need to change
> >> over time, and it’s important that the KIP author(s) update it
> accordingly.
> >>
> >> Just a thought.
> >>
> >> Thanks,
> >> Kirk
> >>
> >>> On Jul 31, 2024, at 9:43 AM, Josep Prat 
> >> wrote:
> >>>
> >>> Hi Colin,
> >>>
> >>> I'm not attached to any names, I'm happy to change them.
> >>> But from what I can guess from KIP-848 the order is EA -> Preview -> GA
> >>> Maybe I'm mistaken.
> >>>
> >>> This is why this discussion is important so we all agree on the number
> of
> >>> steps, order and names.
> >>>
> >>> I know finding a universal solution is maybe impossible but if we can
> >> find
> >>> one that fits most of the feature releases we had and we are planning
> >>> currently would be great.
> >>>
> >>> Best,
> >>>
> >>> --
> >>> Josep Prat
> >>> Open Source Engineering Director, Aiven
> >>> josep.p...@aiven.io   |   +491715557497 | aiven.io
> >>> Aiven Deutschland GmbH
> >>> Alexanderufer 3-7, 10117 Berlin
> >>> Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen,
> >>> Anna Richardson, Kenneth Chen
> >>> Amtsgericht Charlottenburg, HRB 209739 B
> >>>
> >>> On Wed, Jul 31, 2024, 18:31 Colin McCabe  wrote:
> >>>
>  Hi Josep,
> 
>  Thanks for bringing up this discussion.
> 
>  My impression was that "preview" was earlier than early access.
> 
>  In my mind, "preview" somewhat implies that the feature could be
>  incomplete or partial. For example, if someone said their game was a
>  "preview" I would expect kind of a demo.
> 
>  Early access (EA) means people can "access" it but it's "early". In
> >> other
>  words, use at your own risk.
> 
>  What you are calling "production ready," I think we've sometimes
> called
>  generally available (GA) in the past. I don't have a strong preference
> >> on
>  te

[jira] [Created] (KAFKA-17233) MirrorCheckpointConnector should use batched listConsumerGroupOffsets

[Greg Harris (Jira)]

Greg Harris created KAFKA-17233:
---

 Summary: MirrorCheckpointConnector should use batched 
listConsumerGroupOffsets
 Key: KAFKA-17233
 URL: https://issues.apache.org/jira/browse/KAFKA-17233
 Project: Kafka
  Issue Type: Bug
  Components: mirrormaker
Affects Versions: 3.9.0
Reporter: Greg Harris


The AdminClient provides us a single-group listConsumerGroupOffsets, and a 
batched listConsumerGroupOffsets. Because we are potentially requesting the 
offsets for hundreds or thousands of groups, serially requesting all of the 
offsets may be very slow.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: New release branch 3.9

[Chris Egerton]

Hi Colin,

Would it be alright if I cherry-picked
https://github.com/apache/kafka/pull/16678 onto 3.9? This isn't a
regression but it's a low-risk fix that I'd like to get into the next
release if possible.

Cheers,

Chris

On Wed, Jul 31, 2024 at 12:29 AM Ismael Juma  wrote:

> I would recommend against large refactorings in trunk until the first RC
> for 3.9 - that will reduce cherry-pick friction. Once we have the first RC,
> subsequent changes to 3.9 should be limited in scope.
>
> Ismael
>
> On Tue, Jul 30, 2024 at 4:31 PM Colin McCabe  wrote:
>
> > Yeah, please go ahead. I know a lot of people are waiting for 4.0.
> >
> > best,
> > Colin
> >
> >
> > On Tue, Jul 30, 2024, at 16:05, Matthias J. Sax wrote:
> > > Thanks for clarifying Colin. So my assumptions were actually correct.
> > >
> > > We have a lot of contributors waiting to pick-up 4.0 tickets, and I'll
> > > go ahead a tell them that we are ready and they can start to pick them
> > up.
> > >
> > > Thanks.
> > >
> > >
> > > -Matthias
> > >
> > > On 7/30/24 3:51 PM, Colin McCabe wrote:
> > >> Hi Chia-Ping Tsai,
> > >>
> > >> If you can get them done this week then I think we can merge them in
> to
> > 3.9. If not, then let's wait until 4.0, please.
> > >>
> > >> best,
> > >> Colin
> > >>
> > >>
> > >> On Tue, Jul 30, 2024, at 09:07, Chia-Ping Tsai wrote:
> > >>> hi Colin,
> > >>>
> > >>> Could you please consider adding
> > >>> https://issues.apache.org/jira/browse/KAFKA-1 to 3.9.0
> > >>>
> > >>> The issue is used to deprecate the formatters in core module. Also,
> it
> > >>> implements the replacements for them.
> > >>>
> > >>> In order to follow the deprecation rules, it would be nice to have
> > >>> KAFKA-1 in 3.9.0
> > >>>
> > >>> If you agree to have them in 3.9.0, I will cherry-pick them into
> 3.9.0
> > when
> > >>> they get merged to trunk.
> > >>>
> > >>> Best,
> > >>> Chia-Ping
> > >>>
> > >>>
> > >>> José Armando García Sancio  於
> > 2024年7月30日 週二
> > >>> 下午11:59寫道：
> > >>>
> >  Thanks Colin.
> > 
> >  For KIP-853 (KRaft Controller Membership Changes), we still have the
> >  following features that are in progress.
> > 
> >  1. UpdateVoter RPC and request handling
> >  
> >  2. Storage tool changes for KIP-853
> >  
> >  3. kafka-metadata-quorum describe changes for KIP-853
> >  
> >  4. kafka-metadata-quorum add voter and remove voter changes
> >  
> >  5. Sending UpdateVoter request and response handling
> >  
> > 
> >  Can we cherry pick them to the release branch 3.9.0 when they get
> > merged to
> >  trunk? They have a small impact as they shouldn't affect the rest of
> > Kafka
> >  and only affect the kraft controller membership change feature. I
> > expected
> >  them to get merged to the trunk branch in the coming days.
> > 
> >  Thanks,
> > 
> >  On Mon, Jul 29, 2024 at 7:02 PM Colin McCabe 
> > wrote:
> > 
> > > Hi Kafka developers and friends,
> > >
> > > As promised, we now have a release branch for the upcoming 3.9.0
> > release.
> > > Trunk has been bumped to 4.0.0-SNAPSHOT.
> > >
> > > I'll be going over the JIRAs to move every non-blocker from this
> > release
> >  to
> > > the next release.
> > >
> > >  From this point, most changes should go to trunk.
> > > *Blockers (existing and new that we discover while testing the
> > release)
> > > will be double-committed. *Please discuss with your reviewer
> whether
> > your
> > > PR should go to trunk or to trunk+release so they can merge
> > accordingly.
> > >
> > > *Please help us test the release! *
> > >
> > > best,
> > > Colin
> > >
> > 
> > 
> >  --
> >  -José
> > 
> >
>

Jenkins build is unstable: Kafka » Kafka Branch Builder » trunk #3156

[Apache Jenkins Server]

See 

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 3.9 #7

[Apache Jenkins Server]

See 

Re: [DISCUSS] KIP-877: Mechanism for plugins and connectors to register metrics

[Tom Bentley]

Hi Mickael,

I took another look at this KIP (prompted by your bumping of the vote
thread) and had a few more points:

1. `PluginMetrics.metricName()` says "Tags to uniquely identify the plugins
are automatically added to the provided tags". This would seem to imply
that either some tags may not be passed by clients of this interface, or
will be ignored because they collide with ones which are automatically
added, or the automatically added ones can be overridden, voiding the
uniqueness guarantee. I think we should say what the policy is, ideally in
the Javadoc.

2. The proposed name for a converter is
"kafka.connect:type=plugins,connector=my-sink,task=0,iskey=true". Using
"iskey=true" means there's nothing that directly hints that this metric is
from a converter (compare with predicates and transactions). If you didn't
know that the `iskey` tag was used for converters you might not guess --
the metric name suggests only that the metrics are to do with a task and a
key, which doesn't necessarily imply converter. Did you consider using
`converter=key` and `converter=value`?

3. "The PluginMetrics interface implements Closeable, and the close()
method deletes all the metrics the instance has created. It is
automatically closed when the associated instance is closed by the
component that owns it." Do we need to expose close() to users through the
interface? It raises the possibility that they close() it and then try to
register new metrics. If we don't have a use case in mind where a plugin
actually needs to remove all its metrics before the plugin itself is closed
it might be better to leave close() on the implementation only, to avoid
having to cope with this close-then-add possibility.

Sorry for the late feedback.

Tom

On Sat, 1 Jun 2024 at 02:45, Chris Egerton  wrote:

> Hi Mickael,
>
> Apologies for the delay, and many thanks for the updates to the KIP. I'm
> happy with the proposal to automatically close metrics for plugins, even
> though it does come at the cost of dropping the new
> AbstractConfig::getConfiguredInstance and
> AbstractConfig::getConfiguredInstances variants. If the internal-only API
> is smooth enough, perhaps we can add it to the public API in a follow-up
> KIP.
>
> I'm happy with this and am ready to vote in favor.
>
> Cheers,
>
> Chris
>
> On Fri, May 31, 2024 at 10:02 AM Mickael Maison 
> wrote:
>
> > Bumping this thread.
> >
> > If there are no other comments, I'll restart a vote in the next couple of
> > weeks.
> >
> > Thanks,
> > Mickael
> >
> > On Thu, Apr 25, 2024 at 3:28 PM Mickael Maison  >
> > wrote:
> > >
> > > Hi Greg,
> > >
> > > Thanks for taking a close look at the KIP.
> > >
> > > 1/2) I understand your concern about leaking resources. I've played a
> > > bit more with the code and I think we should be able to handle the
> > > closing of the metrics internally rather than delegating it to the
> > > user code. I built a small PoC inspired by your MonitorablePlugin
> > > class example and that looked fine. I think we can even keep that
> > > class internal. I updated the KIP accordingly.
> > >
> > > 3) An earlier version of the proposal used connector and task contexts
> > > to allow them to retrieve their PluginMetrics instance. In a previous
> > > comment Chris suggested switching to implementing Monitorable for
> > > consistency. I think both approaches have pros and cons. I agree with
> > > you that implementing Monitorable with cause compatibility issues with
> > > older Connect runtimes. For that reason, I'm leaning towards
> > > reintroducing the context mechanism. However we would still have this
> > > issue with Converters/Transformations/Predicates. I think it's
> > > typically a bit less problematic with these plugins but it's worth
> > > considering the different approaches. If we can't agree on an approach
> > > we can exclude Connect from this proposal and revisit it at a later
> > > point.
> > >
> > > 4) If this KIP is accepted, I plan to follow up with another KIP to
> > > make MirrorMaker use this mechanism instead of the custom metrics
> > > logic it currently uses.
> > >
> > > Thanks,
> > > Mickael
> > >
> > >
> > >
> > >
> > > On Wed, Apr 24, 2024 at 9:03 PM Mickael Maison <
> mickael.mai...@gmail.com>
> > wrote:
> > > >
> > > > Hi Matthias,
> > > >
> > > > I'm not sure making the Monitorable interface Closeable really solves
> > the issue.
> > > > Ultimately you need to understand the lifecycle of a plugin to
> > > > determine when it make sense to close it and which part of the code
> is
> > > > responsible for doing it. I'd rather have this described properly in
> > > > the interface of the plugin itself than it being a side effect of
> > > > implementing Monitorable.
> > > >
> > > > Looking at Streams, as far as I can tell the only pluggable
> interfaces
> > > > that are Closeable today are the Serdes. It seems Streams can accept
> > > > Serdes instances created by the user [0]. In that case, I think it's
> > > > probably best to ignore Streams

[jira] [Created] (KAFKA-17234) Partition.tryCompleteDelayedRequests should not throw

[Colin McCabe (Jira)]

Colin McCabe created KAFKA-17234:


 Summary: Partition.tryCompleteDelayedRequests should not throw 
 Key: KAFKA-17234
 URL: https://issues.apache.org/jira/browse/KAFKA-17234
 Project: Kafka
  Issue Type: Bug
Reporter: Colin McCabe
Assignee: Colin McCabe






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] Graduation steps for Features

[Matthias J. Sax]

Hi,

thanks for starting this discussion. It's for sure useful.

I don't care too much about the naming personally, it just should be 
well defined.


I agree to Josep, that I don't think we would need to change the KIP 
template. A KIP is a design doc for the full feature, and should not 
concern itself with what state is covered in which release. (As a matter 
of fact, most KIPs don't even include their release version -- we only 
track it in the top-level KIP overview page...)


It might be useful though, to add information about different stages in 
the KIP overview page, which atm only says "released in version 3.8" 
what seems not to be fined grained enough, and hard to reason about. For 
something like KIP-848 this should be simpler to achieve as it's single 
KIP. For KRaft which actually is a high level umbrella KIP with many 
child KIPs it's much harder to reason about it though. Maybe the table 
we use right now it not the right way for stuff like this and we should 
just track this differently (not sure right now how to be fair).


Keeping entry level low, seems not to be a concern, given that new 
contributes would most likely work on smaller KIPs which are completed 
in a single release and thus don't require a more complex workflow and 
tracking.



-Matthias

On 7/31/24 12:25 PM, Josep Prat wrote:

Hi,

I think we can keep the KIP template as is and have the extra process in
the Jira area.

EA, Preview and Production Ready (I also kind of like this better) feel to
me more project management while the KIP document itself is more about the
nature of the changes and describing the end result. Except for some KIPs
that inherently describe a process delivered in stages.

Best,
--
Josep Prat
Open Source Engineering Director, Aiven
josep.p...@aiven.io   |   +491715557497 | aiven.io
Aiven Deutschland GmbH
Alexanderufer 3-7, 10117 Berlin
Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen,
Anna Richardson, Kenneth Chen
Amtsgericht Charlottenburg, HRB 209739 B

On Wed, Jul 31, 2024, 21:12 Andrew Schofield 
wrote:


Hi,
I would at least like to see agreement on the various stages. I’m used to
Early Access, then Preview, and finally General Availability, but I notice
that
at least one other person on this thread had the order different. Also,
General Availability sounds a bit wrong for an open-source project, so
maybe
Production Ready is better.

I’d also like to see some guidelines for what to expect for features in the
various stages. For example, prior to Production Ready, I would not expect
a feature to be turned on by default for a new broker.

The point about barrier to entry is fair, but I think this really only
applies to
a handful of large KIPs which take several releases to come to fruition.
I’m sure we didn’t know when KRaft began which release would be the first
in which it would be production ready. I’m sure we decided to release some
early
code long before that and having a proper name for that stage seems
appropriate.

Thanks,
Andrew


On 31 Jul 2024, at 18:30, Chris Egerton  wrote:

I'd be hesitant to add too much to the KIP template. We should be careful
to keep the barrier to entry low for new users.

On Wed, Jul 31, 2024, 13:24 Kirk True  wrote:


Hi Josep,

Thanks for starting this conversation!

Yes, for KIPs that span multiple releases, clearly defining the KIP’s
state for each release is very important. Perhaps the KIP template

could be

updated to include a “Release Plan” section that includes entries that
include:

- Version: future Apache Kafka release
  - Status: one of Preview, Early access, Production, etc.
  - Jira: umbrella Jira under which other Jiras relevant to that release
are kept
  - Notes: Free-form notes as needed by KIP author

Using KIP-848 as an example, the “Release Plan” section could look like
this:

- 3.7
  - Status: Early access
  - Jira: KAFKA-123
  - Notes: includes support for user-assigned partitions but not
subscriptions, server-side regex not included
- 3.8
  - Status: Preview
  - Jira: KAFKA-234
  - Notes: includes support for subscriptions and lots of bug fixes
- 4.0
  - Status: Production
  - Jira: KAFKA-345
  - Notes: includes feature parity with “CLASSIC” consumer groups

Of course, the contents of the “Release Plan” section may need to change
over time, and it’s important that the KIP author(s) update it

accordingly.


Just a thought.

Thanks,
Kirk


On Jul 31, 2024, at 9:43 AM, Josep Prat 

wrote:


Hi Colin,

I'm not attached to any names, I'm happy to change them.
But from what I can guess from KIP-848 the order is EA -> Preview -> GA
Maybe I'm mistaken.

This is why this discussion is important so we all agree on the number

of

steps, order and names.

I know finding a universal solution is maybe impossible but if we can

find

one that fits most of the feature releases we had and we are planning
currently would be great.

Best,

--
Josep Prat
Open Source Engineering Director, Aiven
josep.p...@aiven.io   |  

Re: [DISCUSS] KIP-512: make Record Headers available in onAcknowledgement

[Rich C.]

Hi Matthias,

> (100) To me, Headers store "application metadata" of a record, but they
are not "Kafka native" record metadata

fair enough.

> (200) extending `onComplete()` and pass in the Headers might be a good
thing to do, too

I agree it’s a nice feature to have. We can increase the scope for now
since option 1 already addresses it. If the community agrees on
`onAcknowledgement` but not on `onCompletion`, we can adjust the scope
accordingly. I apologize for using the term "edge case"; I would consider
it a special case.

> (200) [side mark] we should rather change the interceptor interface and
change the `onSend()` return type to `void`

This is out of scope, but personally, I think returning a new record gives
developers flexibility. Imagine a ProducerInterceptor mutating two fields.
Allowing the return of a new record makes it an all-or-nothing operation.
If a developer chooses to mutate directly, the first mutation might apply
successfully while the second raises an exception, resulting in a partial
state success. Additionally, there are other reasons, such as thread safety.
https://www.perplexity.ai/search/why-producerinterceptor-return-XoVscdipTYCtcXUgx6Fbyg

> (300) there could be even other use-case which might benefit from an easy
access to Headers in the callbacks

I totally agree. Even for latency measurement, I just mentioned to Lianet
that headers are more generic and helpful in other end-to-end latency
scenarios.

> (400) If we really go with overloading `onAcknowledgement()` (and maybe
also `onCompletion()`), and wondering if we should deprecate the existing
overloads?

I prefer to keep the old interface. This decision is intended to help
reduce the work required for client upgrades.

Regards,
Rich


On Tue, Jul 30, 2024 at 9:48 PM Matthias J. Sax  wrote:

> Thanks for updating the KIP.
>
> (100) As stated previously, I personally don't think that adding Headers
> to RecordMetadata is the right thing to do. To me, Headers store
> "application metadata" of a record, but they are not "Kafka native"
> record metadata (ie metadata Kafka an reason about). Headers are a black
> box to Kafka, similar to key and value.
>
> The JavaDocs of `RecordMetadata` might not be helpful, but I agree with
> Andrew and Lianet that its original purpose does not fit to the idea to
> add Headers to it.
>
>
>
> (200) However, I don't understand the argument about `onComplete()`? Of
> course, if the interceptor `onSend()` methods modified the headers,
> `onComplete()` would see the modified record, no the original one. But
> the same should be true for `onAcknowledgement()` (no matter if we pass
> the Headers as parameter of via `RecordMetadata`), right?
>
> Thus, I would personally argue, that extending `onComplete()` and pass
> in the Headers might be a good thing to do, too.
>
>
>
> [side remark] I personally don't agree to the comment on the linked PR
> (even if this is not relevant to this KIP):
>
> > it is not recommended but allowable to create a new ProducerRecord in
> Interceptor.
>
> In the end, `onSend()` has return type `ProducerRecord`, and while I
> agree that it should be used with care, it seems odd to call it an "edge
> case", and position it at something that is only "tolerated"... In the
> end, if we really want to consider modifying a record `onSend()` as bad
> practice, we should rather change the interceptor interface and change
> the `onSend()` return type to `void`.
>
>
>
> (300) About Lianet's idea to just track "producer local timestamp" and
> pass into the callback: I find it interesting, but I am not totally sure
> if we might make it too complicated?
>
> In general, I believe that we should actually have a much larger change
> to the Kafka message format, and always store the producer provided
> "create timestamp" plus the broker side "log append timestamp". For this
> case, `RecordMetadata` would be extended to also provide both
> timestamps. Of course, changing the message format is totally
> out-of-scope for this KIP, and I don't propose to tackle it with this
> KIP. However, if we would want to be forward looking (and optimistically
> assume that we might change the message format at some point in the
> future accordingly), we could actually deprecate the existing
> `RecordMetadata#timestamp()` method, and add `#createTimestamp()` and
> `#logAppendTimestamp()` methods, and clearly document their semantics
> with regard to the corresponding "CreateTime" vs "AppendTime" topic config.
>
> The disadvantage I see is, that we do something that might never happen
> in the Kafka message format, and that we would limit the scope of this
> KIP to the "measure latency" use-case only.
>
> The KIP also mentioned tracing as use-case which could benefit to access
> Headers in the callback what I find convincing. I don't see a good
> reason why we would want to exclude this use-case? -- And frankly, there
> could be even other use-case which might benefit from an easy access to
> Headers in the 

Inquiry on Compact Topic Behavior in Kafka 3.7.0

[Rich C.]

Hi, dev community,

We are currently working with Kafka 3.7.0 and have configured a topic with
the following settings:

log.cleanup.policy=compact
segment.ms=12 (2 minutes)
log.roll.hours (default 7 days)

This topic is used for testing purposes and has a low message volume.
Specifically, it contains a single partition with two messages having the
same key, separated by 5 minutes. After these messages, no new messages
have been added for the past 10 days.

*Expected behavior*: Both messages should be compacted into one, given the
log.cleanup.policy=compact.

*Actual behavior*: Both messages remain on the topic, and no compaction
seems to have occurred.

I have investigated the code and observed a scheduled job for the delete
policy that checks every 5 minutes and closes active segments if retention
criteria are met. However, I did not find a similar scheduler for compact
topics.

Could you please clarify if this behavior is expected under the current
configuration? If it is not, might this be a potential bug? Additionally,
am I possibly overlooking any configurations or mechanisms related to
compaction that could explain this behavior?

Thank you for your assistance.

Regards,
Rich

Re: [DISCUSS] Graduation steps for Features

[Chris Egerton]

I only raised the idea of barrier to entry regarding the template. I did
not state or even really imply that it should affect this initiative in
general. Please do not feel obligated to discuss that concept further if
we're all on the same page RE not adding to the template.

On Wed, Jul 31, 2024, 22:02 Matthias J. Sax  wrote:

> Hi,
>
> thanks for starting this discussion. It's for sure useful.
>
> I don't care too much about the naming personally, it just should be
> well defined.
>
> I agree to Josep, that I don't think we would need to change the KIP
> template. A KIP is a design doc for the full feature, and should not
> concern itself with what state is covered in which release. (As a matter
> of fact, most KIPs don't even include their release version -- we only
> track it in the top-level KIP overview page...)
>
> It might be useful though, to add information about different stages in
> the KIP overview page, which atm only says "released in version 3.8"
> what seems not to be fined grained enough, and hard to reason about. For
> something like KIP-848 this should be simpler to achieve as it's single
> KIP. For KRaft which actually is a high level umbrella KIP with many
> child KIPs it's much harder to reason about it though. Maybe the table
> we use right now it not the right way for stuff like this and we should
> just track this differently (not sure right now how to be fair).
>
> Keeping entry level low, seems not to be a concern, given that new
> contributes would most likely work on smaller KIPs which are completed
> in a single release and thus don't require a more complex workflow and
> tracking.
>
>
> -Matthias
>
> On 7/31/24 12:25 PM, Josep Prat wrote:
> > Hi,
> >
> > I think we can keep the KIP template as is and have the extra process in
> > the Jira area.
> >
> > EA, Preview and Production Ready (I also kind of like this better) feel
> to
> > me more project management while the KIP document itself is more about
> the
> > nature of the changes and describing the end result. Except for some KIPs
> > that inherently describe a process delivered in stages.
> >
> > Best,
> > --
> > Josep Prat
> > Open Source Engineering Director, Aiven
> > josep.p...@aiven.io   |   +491715557497 | aiven.io
> > Aiven Deutschland GmbH
> > Alexanderufer 3-7, 10117 Berlin
> > Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen,
> > Anna Richardson, Kenneth Chen
> > Amtsgericht Charlottenburg, HRB 209739 B
> >
> > On Wed, Jul 31, 2024, 21:12 Andrew Schofield 
> > wrote:
> >
> >> Hi,
> >> I would at least like to see agreement on the various stages. I’m used
> to
> >> Early Access, then Preview, and finally General Availability, but I
> notice
> >> that
> >> at least one other person on this thread had the order different. Also,
> >> General Availability sounds a bit wrong for an open-source project, so
> >> maybe
> >> Production Ready is better.
> >>
> >> I’d also like to see some guidelines for what to expect for features in
> the
> >> various stages. For example, prior to Production Ready, I would not
> expect
> >> a feature to be turned on by default for a new broker.
> >>
> >> The point about barrier to entry is fair, but I think this really only
> >> applies to
> >> a handful of large KIPs which take several releases to come to fruition.
> >> I’m sure we didn’t know when KRaft began which release would be the
> first
> >> in which it would be production ready. I’m sure we decided to release
> some
> >> early
> >> code long before that and having a proper name for that stage seems
> >> appropriate.
> >>
> >> Thanks,
> >> Andrew
> >>
> >>> On 31 Jul 2024, at 18:30, Chris Egerton 
> wrote:
> >>>
> >>> I'd be hesitant to add too much to the KIP template. We should be
> careful
> >>> to keep the barrier to entry low for new users.
> >>>
> >>> On Wed, Jul 31, 2024, 13:24 Kirk True  wrote:
> >>>
>  Hi Josep,
> 
>  Thanks for starting this conversation!
> 
>  Yes, for KIPs that span multiple releases, clearly defining the KIP’s
>  state for each release is very important. Perhaps the KIP template
> >> could be
>  updated to include a “Release Plan” section that includes entries that
>  include:
> 
>  - Version: future Apache Kafka release
>    - Status: one of Preview, Early access, Production, etc.
>    - Jira: umbrella Jira under which other Jiras relevant to that
> release
>  are kept
>    - Notes: Free-form notes as needed by KIP author
> 
>  Using KIP-848 as an example, the “Release Plan” section could look
> like
>  this:
> 
>  - 3.7
>    - Status: Early access
>    - Jira: KAFKA-123
>    - Notes: includes support for user-assigned partitions but not
>  subscriptions, server-side regex not included
>  - 3.8
>    - Status: Preview
>    - Jira: KAFKA-234
>    - Notes: includes support for subscriptions and lots of bug fixes
>  - 4.0
>    - Status: Production
>    - Jira: KAFKA-3

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 3.9 #8

[Apache Jenkins Server]

See 

[jira] [Created] (KAFKA-17235) system test test_performance_service.py failed

[Kuan Po Tseng (Jira)]

Kuan Po Tseng created KAFKA-17235:
-

 Summary: system test test_performance_service.py failed
 Key: KAFKA-17235
 URL: https://issues.apache.org/jira/browse/KAFKA-17235
 Project: Kafka
  Issue Type: Bug
  Components: system tests
Reporter: Kuan Po Tseng
Assignee: Kuan Po Tseng


After run
{code:java}
TC_PATHS="tests/kafkatest/sanity_checks/test_performance_services.py" bash 
tests/docker/run_tests.sh
{code}
I got error
{code:java}
Exception in thread "main" java.lang.NoSuchFieldError: SYSTEM
at 
org.apache.kafka.tools.PushHttpMetricsReporter.(PushHttpMetricsReporter.java:99)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at 
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at 
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at java.lang.Class.newInstance(Class.java:442)
at org.apache.kafka.common.utils.Utils.newInstance(Utils.java:231)
at 
org.apache.kafka.common.config.AbstractConfig.getConfiguredInstances(AbstractConfig.java:156)
at 
org.apache.kafka.clients.producer.KafkaProducer.(KafkaProducer.java:168)
at 
org.apache.kafka.clients.producer.KafkaProducer.(KafkaProducer.java:129)
at 
org.apache.kafka.tools.ProducerPerformance.createKafkaProducer(ProducerPerformance.java:175)
at 
org.apache.kafka.tools.ProducerPerformance.start(ProducerPerformance.java:86)
at 
org.apache.kafka.tools.ProducerPerformance.main(ProducerPerformance.java:52){code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-17236) local log should be deleted when remote copy disabled and no remote logs

[Luke Chen (Jira)]

Luke Chen created KAFKA-17236:
-

 Summary: local log should be deleted when remote copy disabled and 
no remote logs
 Key: KAFKA-17236
 URL: https://issues.apache.org/jira/browse/KAFKA-17236
 Project: Kafka
  Issue Type: Sub-task
Reporter: Luke Chen
Assignee: Luke Chen


local log should be deleted when remote copy disabled and no remote logs. It 
doesn't work right now because the remote storage is enabled but no more data 
will be copied into remote storage.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)