[jira] [Created] (KAFKA-18906) Upgrade to Junit 5.13 and apply ParameterizedClass

2025-03-01 Thread Chia-Ping Tsai (Jira)
Chia-Ping Tsai created KAFKA-18906:
--

 Summary: Upgrade to Junit 5.13 and apply ParameterizedClass
 Key: KAFKA-18906
 URL: https://issues.apache.org/jira/browse/KAFKA-18906
 Project: Kafka
  Issue Type: Test
Reporter: Chia-Ping Tsai
Assignee: Chia-Ping Tsai


Junit 5.13 will include the feature: {{ParameterizedClass}} which allows use to 
define parameters on the class. That can remove many duplicate annotations from 
tests.

ref: https://github.com/junit-team/junit5/issues/878



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Created] (KAFKA-18908) the size of appended value can't be larger than Short.MAX_VALUE

2025-03-01 Thread Chia-Ping Tsai (Jira)
Chia-Ping Tsai created KAFKA-18908:
--

 Summary: the size of appended value can't be larger than 
Short.MAX_VALUE
 Key: KAFKA-18908
 URL: https://issues.apache.org/jira/browse/KAFKA-18908
 Project: Kafka
  Issue Type: Sub-task
Reporter: Chia-Ping Tsai
Assignee: Chia-Ping Tsai


related to https://issues.apache.org/jira/browse/KAFKA-18907

we should document it for 4.0



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Resolved] (KAFKA-17981) add Integration test for ConfigCommand to add config `key=[val1,val2]`

2025-03-01 Thread Chia-Ping Tsai (Jira)


 [ 
https://issues.apache.org/jira/browse/KAFKA-17981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chia-Ping Tsai resolved KAFKA-17981.

Fix Version/s: 4.1.0
   Resolution: Fixed

> add Integration test for ConfigCommand to add config `key=[val1,val2]`
> --
>
> Key: KAFKA-17981
> URL: https://issues.apache.org/jira/browse/KAFKA-17981
> Project: Kafka
>  Issue Type: Bug
>Reporter: Chia-Ping Tsai
>Assignee: Nick Guo
>Priority: Minor
> Fix For: 4.1.0
>
>
> follow-up of KAFKA-17583



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


Re: [DISCUSS] KIP-1124: Providing a clear Kafka Client upgrade path

2025-03-01 Thread Chia-Ping Tsai
hi

> If this Connect case is the only one that deviates from that rule, I would
recommended reverting it. But if there are many more cases, we may need to
look into them in more detail.

I briefly checked the commit history, and this is the only one that needs
to be highlighted. Fortunately, it is easy to revert.

> why was it removed? Oversight (ie, accidentally
too early) or deliberately?

deliberately.

I have left a common on the PR (
https://github.com/apache/kafka/pull/17412#issuecomment-2692600494)

Best,
Chia-Ping


Matthias J. Sax  於 2025年3月2日 週日 上午6:32寫道:

> Don't have a strong opinion about the Connect case.
>
> I guess the question is: why was it removed? Oversight (ie, accidentally
> too early) or deliberately? Also, what is the impact if we keep the API
> and revert the removal?
>
> IMHO, the 3-releases-rule can have exceptions, if there are good reason
> for it. I just don't have enough context about the change to judge.
>
>
>
> >> One more thing, 3.7 was released in Feb 2024 and hence it will be more
> than
> >> 12 months by the time 4.0 is released. Technically, it would be ok to
> >> remove APIs deprecated in 3.7.
>
> Not sure if this is a strong argument. We cannot time releases totally
> accurately, so the 3-prior-releases rule should be the "source of truth"
> IMHO, that just practically translated to "about 1 year". But as I said
> above, I also don't think we need to be dogmatic about it.
>
>
>
> -Matthias
>
> On 3/1/25 8:37 AM, Ismael Juma wrote:
> > One more thing, 3.7 was released in Feb 2024 and hence it will be more
> than
> > 12 months by the time 4.0 is released. Technically, it would be ok to
> > remove APIs deprecated in 3.7.
> >
> > Ismael
> >
> > On Sat, Mar 1, 2025, 8:34 AM Ismael Juma  wrote:
> >
> >> Hi Chia-Ping,
> >>
> >> Is this Connect removal the only one that did not meet the stated
> >> requirement?
> >>
> >> We do indeed aim not to remove APIs deprecated within the last 12 months
> >> before the next major release, but I'm not sure how strictly this is
> >> followed (since it's not enforced).
> >>
> >> If this Connect case is the only one that deviates from that rule, I
> would
> >> recommended reverting it. But if there are many more cases, we may need
> to
> >> look into them in more detail.
> >>
> >> Ismael
> >>
> >> On Fri, Feb 28, 2025, 7:10 PM Chia-Ping Tsai 
> wrote:
> >>
> >>> hi Matthias
> >>>
> >>> thanks for your updates. it looks great and readable.
> >>>
> >>> Regarding API compatibility, we state "we only provide API backward
> >>> compatibility for version 3.7, 3.8, and 3.9" However, within Connect
> APIs,
> >>> we have removed a public API that was deprecated by 3.7 [0][1].
> >>>
> >>> Since this KIP will likely include Connect APIs, it's important to
> >>> discuss this now. We could: 1) add extra descriptions to highlight the
> >>> difference for Connect APIs, or 2) revert the commit that removed the
> >>> public API for consistent documentation.
> >>>
> >>> WDYT?
> >>>
> >>> [0]
> >>>
> https://github.com/apache/kafka/commit/1c8bb61a43d3ad1fd7a10eb3947342ceba783c4e
> >>> [1]
> >>>
> https://github.com/apache/kafka/commit/4f1688742e75a147741fbdc307c5e85d2addb811
> >>>
> >>> Best,
> >>> Chia-Ping
> >>>
> >>> On 2025/02/28 22:44:22 "Matthias J. Sax" wrote:
>  Thanks for pointing me to test Ismael. Glad to see I just missed it,
> >>> and
>  that we indeed have a test for it.
> 
> 
> >> Feel free to edit the KIP directly – I'm totally fine with it.
> >>> Thanks a lot!
> 
>  Thanks Kuan-Po!
> 
>  I did update the wiki page and added more details. I did follow
> Bruno's
>  suggestion, to be a little bit more elaborate, as the KIP page is
> >>> mainly
>  for Kafka maintainers. -- We might want to do it differently in user
>  facing docs, but detailed doc updates seems to be out-of-scope for the
>  KIP discussion and we can take it on the corresponding PR, to simplify
>  it further.
> 
> 
>  I did keep the 2.8.x bridge release for KS for now.; we can still
> >>> change
>  the recommendation and skip 2.8.x as bridge release of course, if we
>  think it would be better for users.
> 
> 
>  Curious the get your feedback.
> 
> 
>  -Matthias
> 
> 
>  On 2/28/25 8:15 AM, Kuan Po Tseng wrote:
> > Hi Matthias,
> >
> > Feel free to edit the KIP directly – I'm totally fine with it. Thanks
> >>> a lot!
> >
> > Best,
> > Kuan-Po Tseng
> >
> > On 2025/02/28 07:35:10 "Matthias J. Sax" wrote:
> >> Thanks all. I am happy that we are making progress to close this
> out.
> >>
> >> As mentioned in my (long) email last Friday, I also think that the
> >>> KIP
> >> makes a lot of implicit assumptions, and it might benefit from begin
> >> more explicit, even explaining things that are "set in stone"
> >>> already,
> >> and the KIP does not change them (like API deprecation timelines and
> >> guarantees). Bu

Re: [DISCUSS] KIP-1124: Providing a clear Kafka Client upgrade path

2025-03-01 Thread Matthias J. Sax

Don't have a strong opinion about the Connect case.

I guess the question is: why was it removed? Oversight (ie, accidentally 
too early) or deliberately? Also, what is the impact if we keep the API 
and revert the removal?


IMHO, the 3-releases-rule can have exceptions, if there are good reason 
for it. I just don't have enough context about the change to judge.





One more thing, 3.7 was released in Feb 2024 and hence it will be more than
12 months by the time 4.0 is released. Technically, it would be ok to
remove APIs deprecated in 3.7.


Not sure if this is a strong argument. We cannot time releases totally 
accurately, so the 3-prior-releases rule should be the "source of truth" 
IMHO, that just practically translated to "about 1 year". But as I said 
above, I also don't think we need to be dogmatic about it.




-Matthias

On 3/1/25 8:37 AM, Ismael Juma wrote:

One more thing, 3.7 was released in Feb 2024 and hence it will be more than
12 months by the time 4.0 is released. Technically, it would be ok to
remove APIs deprecated in 3.7.

Ismael

On Sat, Mar 1, 2025, 8:34 AM Ismael Juma  wrote:


Hi Chia-Ping,

Is this Connect removal the only one that did not meet the stated
requirement?

We do indeed aim not to remove APIs deprecated within the last 12 months
before the next major release, but I'm not sure how strictly this is
followed (since it's not enforced).

If this Connect case is the only one that deviates from that rule, I would
recommended reverting it. But if there are many more cases, we may need to
look into them in more detail.

Ismael

On Fri, Feb 28, 2025, 7:10 PM Chia-Ping Tsai  wrote:


hi Matthias

thanks for your updates. it looks great and readable.

Regarding API compatibility, we state "we only provide API backward
compatibility for version 3.7, 3.8, and 3.9" However, within Connect APIs,
we have removed a public API that was deprecated by 3.7 [0][1].

Since this KIP will likely include Connect APIs, it's important to
discuss this now. We could: 1) add extra descriptions to highlight the
difference for Connect APIs, or 2) revert the commit that removed the
public API for consistent documentation.

WDYT?

[0]
https://github.com/apache/kafka/commit/1c8bb61a43d3ad1fd7a10eb3947342ceba783c4e
[1]
https://github.com/apache/kafka/commit/4f1688742e75a147741fbdc307c5e85d2addb811

Best,
Chia-Ping

On 2025/02/28 22:44:22 "Matthias J. Sax" wrote:

Thanks for pointing me to test Ismael. Glad to see I just missed it,

and

that we indeed have a test for it.



Feel free to edit the KIP directly – I'm totally fine with it.

Thanks a lot!


Thanks Kuan-Po!

I did update the wiki page and added more details. I did follow Bruno's
suggestion, to be a little bit more elaborate, as the KIP page is

mainly

for Kafka maintainers. -- We might want to do it differently in user
facing docs, but detailed doc updates seems to be out-of-scope for the
KIP discussion and we can take it on the corresponding PR, to simplify
it further.


I did keep the 2.8.x bridge release for KS for now.; we can still

change

the recommendation and skip 2.8.x as bridge release of course, if we
think it would be better for users.


Curious the get your feedback.


-Matthias


On 2/28/25 8:15 AM, Kuan Po Tseng wrote:

Hi Matthias,

Feel free to edit the KIP directly – I'm totally fine with it. Thanks

a lot!


Best,
Kuan-Po Tseng

On 2025/02/28 07:35:10 "Matthias J. Sax" wrote:

Thanks all. I am happy that we are making progress to close this out.

As mentioned in my (long) email last Friday, I also think that the

KIP

makes a lot of implicit assumptions, and it might benefit from begin
more explicit, even explaining things that are "set in stone"

already,

and the KIP does not change them (like API deprecation timelines and
guarantees). But as Bruno said, it would help a lot to make it

easier to

understand the KIP.

Only a very small number of people seems to fully understand all the
nuances, and it took quite a while to collect and curate all the

cases,

to shape this KIP... And we know that a lot of people actually read
KIPs, and the title for KIP-1124 is or sure click-bait...

I am also happy to edit the wiki page directly if that's ok with

Kuan,

to add background information that I believe is missing and

important.




My overall understanding of the state of the KIP, ie, its actual

content

seems to be, that there is a high level agreement, and the content of
the KIP is ok.

The only open question seems to be, if we want to recommend `2.8.x`

as a

bridge release for Kafka Streams or not. As said in my email on

Friday,

I am personally ok either way, with a preference to add `2.8.x` as
bridge release, as it simplifies updating the code by avoiding
compilation errors. With a 2.8.x bridge release, KS application code
would _always_ compile, and the JavaDocs help a lot to guide users

how

to migrate off deprecated APIs. -- With a direct upgrade from older
versions to 3.9.x, code does not compile, and there is n

[jira] [Created] (KAFKA-18905) Avoid out of order sequence errors from multiple in-flight batches

2025-03-01 Thread Sean Quah (Jira)
Sean Quah created KAFKA-18905:
-

 Summary: Avoid out of order sequence errors from multiple 
in-flight batches
 Key: KAFKA-18905
 URL: https://issues.apache.org/jira/browse/KAFKA-18905
 Project: Kafka
  Issue Type: Bug
  Components: producer 
Reporter: Sean Quah


Consider a similar setup to KAFKA-9199:

The broker attempts to cache the state of the last 5 batches in order to enable 
duplicate detection. It is possible in some cases for this to result in a 
sequence such as the following:
 # Send batch n
 # Batch n successfully written, but response is delayed
 # Send Batch n+1, receive successful response for n+1
 # Send Batch n+2, receive successful response for n+2
 # Send Batch n+3, receive successful response for n+3
 # Send Batch n+4, receive successful response for n+4
 # Send Batch n+5, receive successful response for n+5
 # Batch n times out, or we receive a {{NOT_LEADER_OR_FOLLOWER}} response
 # Retry batch n, the response is {{OUT_OF_ORDER_SEQUENCE}}, retry again and 
again

This situation could happen with a {{max.in.flight.requests.per.connection}} as 
low as 2.

To avoid this situation, we can avoid putting batch n+5 in flight while batch n 
is in flight.




--
This message was sent by Atlassian Jira
(v8.20.10#820010)


Re: [DISCUSS] KIP-1133: AK Documentation and Website in Markdown

2025-03-01 Thread Matthias J. Sax

Hi,

I am totally in favor to make this happen, and I was disappointed each 
time in the past, when it was proposed but we did not do it.




Given that it is a huge effort, I am wondering if it would be sufficient 
to just to this change for 4.0 release though, and not translate any 
older versions at all?



For including the docs in the release, I am actually wondering what the 
value is? Does anybody actually use these artifact? Of course, we need 
to publish JavaDocs, but the content of the web page seems to have very 
low value to the community? -- I did also check ASF guidelines about it 
and the webpage [1] say:



This material may include documentation concerning the release [...]


So it is optional, but not required to publish docs artifacts from my 
understanding. I have no objections to keep publishing the web page 
docs, just saying, if the effort is too large and the value to low, we 
could consider just dropping it.



I am also in favor of having easy access to `trunk / SNAPSHOT` version 
of the docs if possible. Some other ASF project do this already, for 
example Flink [2]. It's much easier to work with, and allows to spot 
issues during development. While it does not help on a PR directly, it 
is still a good way to verify a PR after it was merged, so any errors 
can be detected and corrected more easily.



Also agree to the other raised points about generated content and 
permalinks. For permalinks, if effort is reasonable, I would rather try 
to redirect as many as possible.




-Matthias




[1] 
https://www.apache.org/legal/release-policy.html#distribute-other-artifacts

[2] https://nightlies.apache.org/flink/flink-docs-master/


On 2/25/25 8:43 AM, David Arthur wrote:

MM1: I suspect Hugo has some way of doing url redirection, but at the very
least we can add redirects in our htaccess file
https://github.com/apache/kafka-site/blob/asf-site/.htaccess

Excluding the "front matter" (intro, quickstart, powered by, etc) I think
the main links we need to worry about are the configuration docs pages. In
the demo, it looks like we are using the same anchors for specific config
sections.

E.g.,
https://kafka-site-md-711970345036.us-central1.run.app/39/configuration/configuration/#topicconfigs_compression.gzip.level
is the same content as
https://kafka.apache.org/documentation/#topicconfigs_compression.zstd.level

So maybe it shouldn't be too hard?

Personally, I think the only links we should consider to be "permalinks"
are the front matter and the documentation pages with the version in the
URL. The unversioned documentation pages (like
https://kafka.apache.org/documentation/#topicconfigs_compression.zstd.level)
can change from release to release, so they are not expected to be
permanent.

Do we have any data on inbound links to the site? Maybe from Google
Analytics or similar?

-David A



On Mon, Feb 24, 2025 at 10:17 AM Mickael Maison 
wrote:


Hi Harish,

This is definitively something we want to do!

MM1. One aspect to keep in mind in terms of compatibility is whether
existing links will still work. Thousands of online resources link to
the Apache Kafka website. Is it possible to keep all links working?

MM2. A bunch of sections are generated. To do so we have a number of
classes in the public API (for example 0, 1) with toHtml() or main()
methods that output HTML. Should we update these to output Markdown
directly instead?

Converting the website has been discussed several times, I'd suggest
checking at least https://issues.apache.org/jira/browse/KAFKA-2967 to
see what was previously said.

0:
https://kafka.apache.org/39/javadoc/org/apache/kafka/common/config/ConfigDef.html#toHtml()
1:
https://kafka.apache.org/39/javadoc/org/apache/kafka/clients/consumer/ConsumerConfig.html#main(java.lang.String%5B%5D)

Thanks,
Mickael

On Mon, Feb 24, 2025 at 3:33 PM Bruno Cadonna  wrote:


Hi Harish,

Thanks for the KIP! Great initiative!

Do you plan to have a development version of the website online in
parallel to the old HTML version?

I understand that locally with the markdown files and an markdown
editor, we see the rough structure of the docs (which is already an
improvement), but we will not see the end product, because for that we
again need a web server.

I assume that after you migrated all docs to markdown, we want to see
the complete website, review the content, and make some refinements.

Best,
Bruno

On 23.02.25 07:52, Harish Vishwanath wrote:

Thank you David. Appreciate your feedback. My comments below:

DA1) I think this is a very good suggestion. While building the

prototype,

I did spend additional effort for older versions of the doc since

there are

fairly significant differences in the structure and format of the

documents

as compared to 3.x versions. I have written the automation which can do
this, but I do think hosting the older versions (<3.x) in its current
format and moving the rest to markdown will reduce the overall
testing/validation effort. That said, if we 

[jira] [Resolved] (KAFKA-18880) Remove kafka.cluster.Broker and BrokerEndPointNotAvailableException

2025-03-01 Thread Chia-Ping Tsai (Jira)


 [ 
https://issues.apache.org/jira/browse/KAFKA-18880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chia-Ping Tsai resolved KAFKA-18880.

Fix Version/s: 4.1.0
   Resolution: Fixed

> Remove kafka.cluster.Broker and BrokerEndPointNotAvailableException
> ---
>
> Key: KAFKA-18880
> URL: https://issues.apache.org/jira/browse/KAFKA-18880
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Chia-Ping Tsai
>Assignee: xuanzhang gong
>Priority: Minor
> Fix For: 4.1.0
>
>
> they are not used anymore after removing zk code



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Resolved] (KAFKA-18868) add the "default value" explanation to the docs of num.replica.alter.log.dirs.threads

2025-03-01 Thread Chia-Ping Tsai (Jira)


 [ 
https://issues.apache.org/jira/browse/KAFKA-18868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chia-Ping Tsai resolved KAFKA-18868.

Resolution: Fixed

trunk: 
[https://github.com/apache/kafka/commit/2ccc73783e30bde27aebb2a26a1f32e1327f6530]

 

4.0: 
https://github.com/apache/kafka/commit/ae798f12be4fb2f770b92d87b0986832e8fd4d82

> add the "default value" explanation to the docs of 
> num.replica.alter.log.dirs.threads
> -
>
> Key: KAFKA-18868
> URL: https://issues.apache.org/jira/browse/KAFKA-18868
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Chia-Ping Tsai
>Assignee: Logan Zhu
>Priority: Trivial
> Fix For: 4.0.0
>
>
> The default value of {{num.replica.alter.log.dirs.threads}} is equal to the 
> number of log directories, but the documentation doesn't mention this. Users 
> only see a "null" default from the doc



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


Re: [DISCUSS] KIP-1124: Providing a clear Kafka Client upgrade path

2025-03-01 Thread Ismael Juma
Hi Chia-Ping,

Is this Connect removal the only one that did not meet the stated
requirement?

We do indeed aim not to remove APIs deprecated within the last 12 months
before the next major release, but I'm not sure how strictly this is
followed (since it's not enforced).

If this Connect case is the only one that deviates from that rule, I would
recommended reverting it. But if there are many more cases, we may need to
look into them in more detail.

Ismael

On Fri, Feb 28, 2025, 7:10 PM Chia-Ping Tsai  wrote:

> hi Matthias
>
> thanks for your updates. it looks great and readable.
>
> Regarding API compatibility, we state "we only provide API backward
> compatibility for version 3.7, 3.8, and 3.9" However, within Connect APIs,
> we have removed a public API that was deprecated by 3.7 [0][1].
>
> Since this KIP will likely include Connect APIs, it's important to discuss
> this now. We could: 1) add extra descriptions to highlight the difference
> for Connect APIs, or 2) revert the commit that removed the public API for
> consistent documentation.
>
> WDYT?
>
> [0]
> https://github.com/apache/kafka/commit/1c8bb61a43d3ad1fd7a10eb3947342ceba783c4e
> [1]
> https://github.com/apache/kafka/commit/4f1688742e75a147741fbdc307c5e85d2addb811
>
> Best,
> Chia-Ping
>
> On 2025/02/28 22:44:22 "Matthias J. Sax" wrote:
> > Thanks for pointing me to test Ismael. Glad to see I just missed it, and
> > that we indeed have a test for it.
> >
> >
> > >> Feel free to edit the KIP directly – I'm totally fine with it. Thanks
> a lot!
> >
> > Thanks Kuan-Po!
> >
> > I did update the wiki page and added more details. I did follow Bruno's
> > suggestion, to be a little bit more elaborate, as the KIP page is mainly
> > for Kafka maintainers. -- We might want to do it differently in user
> > facing docs, but detailed doc updates seems to be out-of-scope for the
> > KIP discussion and we can take it on the corresponding PR, to simplify
> > it further.
> >
> >
> > I did keep the 2.8.x bridge release for KS for now.; we can still change
> > the recommendation and skip 2.8.x as bridge release of course, if we
> > think it would be better for users.
> >
> >
> > Curious the get your feedback.
> >
> >
> > -Matthias
> >
> >
> > On 2/28/25 8:15 AM, Kuan Po Tseng wrote:
> > > Hi Matthias,
> > >
> > > Feel free to edit the KIP directly – I'm totally fine with it. Thanks
> a lot!
> > >
> > > Best,
> > > Kuan-Po Tseng
> > >
> > > On 2025/02/28 07:35:10 "Matthias J. Sax" wrote:
> > >> Thanks all. I am happy that we are making progress to close this out.
> > >>
> > >> As mentioned in my (long) email last Friday, I also think that the KIP
> > >> makes a lot of implicit assumptions, and it might benefit from begin
> > >> more explicit, even explaining things that are "set in stone" already,
> > >> and the KIP does not change them (like API deprecation timelines and
> > >> guarantees). But as Bruno said, it would help a lot to make it easier
> to
> > >> understand the KIP.
> > >>
> > >> Only a very small number of people seems to fully understand all the
> > >> nuances, and it took quite a while to collect and curate all the
> cases,
> > >> to shape this KIP... And we know that a lot of people actually read
> > >> KIPs, and the title for KIP-1124 is or sure click-bait...
> > >>
> > >> I am also happy to edit the wiki page directly if that's ok with Kuan,
> > >> to add background information that I believe is missing and important.
> > >>
> > >>
> > >>
> > >> My overall understanding of the state of the KIP, ie, its actual
> content
> > >> seems to be, that there is a high level agreement, and the content of
> > >> the KIP is ok.
> > >>
> > >> The only open question seems to be, if we want to recommend `2.8.x`
> as a
> > >> bridge release for Kafka Streams or not. As said in my email on
> Friday,
> > >> I am personally ok either way, with a preference to add `2.8.x` as
> > >> bridge release, as it simplifies updating the code by avoiding
> > >> compilation errors. With a 2.8.x bridge release, KS application code
> > >> would _always_ compile, and the JavaDocs help a lot to guide users how
> > >> to migrate off deprecated APIs. -- With a direct upgrade from older
> > >> versions to 3.9.x, code does not compile, and there is no guidance as
> > >> the old methods (and corresponding JavaDocs) are already removed,
> making
> > >> it significant harder for user to update their code and make it
> compile
> > >> again.
> > >>
> > >> In the end, we would only _recommend_ 2.8.x as a bridge release to
> > >> users, and there won't be any claim that an direct upgrade to 3.9.x is
> > >> not supported. If one wants to do this, it's tested, and ok. Knock
> > >> yourself out -- there is for sure apps out there, that are just a
> simple
> > >> `builder.stream(...).filter(...).to(...)` the there won't be an API
> > >> incompatibility issues.
> > >>
> > >> But I don't think it would be wise to _recommend_ a direct upgrade to
> > >> 3.9.x for the reasons stated, as many ap

[jira] [Resolved] (KAFKA-17039) KIP-919 supports for `unregisterBroker`

2025-03-01 Thread Chia-Ping Tsai (Jira)


 [ 
https://issues.apache.org/jira/browse/KAFKA-17039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chia-Ping Tsai resolved KAFKA-17039.

Fix Version/s: 4.1.0
   Resolution: Fixed

> KIP-919 supports for `unregisterBroker`
> ---
>
> Key: KAFKA-17039
> URL: https://issues.apache.org/jira/browse/KAFKA-17039
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Chia-Ping Tsai
>Assignee: TengYao Chi
>Priority: Minor
> Fix For: 4.1.0
>
>
> as title



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


Re: [DISCUSS] KIP-1124: Providing a clear Kafka Client upgrade path

2025-03-01 Thread Ismael Juma
One more thing, 3.7 was released in Feb 2024 and hence it will be more than
12 months by the time 4.0 is released. Technically, it would be ok to
remove APIs deprecated in 3.7.

Ismael

On Sat, Mar 1, 2025, 8:34 AM Ismael Juma  wrote:

> Hi Chia-Ping,
>
> Is this Connect removal the only one that did not meet the stated
> requirement?
>
> We do indeed aim not to remove APIs deprecated within the last 12 months
> before the next major release, but I'm not sure how strictly this is
> followed (since it's not enforced).
>
> If this Connect case is the only one that deviates from that rule, I would
> recommended reverting it. But if there are many more cases, we may need to
> look into them in more detail.
>
> Ismael
>
> On Fri, Feb 28, 2025, 7:10 PM Chia-Ping Tsai  wrote:
>
>> hi Matthias
>>
>> thanks for your updates. it looks great and readable.
>>
>> Regarding API compatibility, we state "we only provide API backward
>> compatibility for version 3.7, 3.8, and 3.9" However, within Connect APIs,
>> we have removed a public API that was deprecated by 3.7 [0][1].
>>
>> Since this KIP will likely include Connect APIs, it's important to
>> discuss this now. We could: 1) add extra descriptions to highlight the
>> difference for Connect APIs, or 2) revert the commit that removed the
>> public API for consistent documentation.
>>
>> WDYT?
>>
>> [0]
>> https://github.com/apache/kafka/commit/1c8bb61a43d3ad1fd7a10eb3947342ceba783c4e
>> [1]
>> https://github.com/apache/kafka/commit/4f1688742e75a147741fbdc307c5e85d2addb811
>>
>> Best,
>> Chia-Ping
>>
>> On 2025/02/28 22:44:22 "Matthias J. Sax" wrote:
>> > Thanks for pointing me to test Ismael. Glad to see I just missed it,
>> and
>> > that we indeed have a test for it.
>> >
>> >
>> > >> Feel free to edit the KIP directly – I'm totally fine with it.
>> Thanks a lot!
>> >
>> > Thanks Kuan-Po!
>> >
>> > I did update the wiki page and added more details. I did follow Bruno's
>> > suggestion, to be a little bit more elaborate, as the KIP page is
>> mainly
>> > for Kafka maintainers. -- We might want to do it differently in user
>> > facing docs, but detailed doc updates seems to be out-of-scope for the
>> > KIP discussion and we can take it on the corresponding PR, to simplify
>> > it further.
>> >
>> >
>> > I did keep the 2.8.x bridge release for KS for now.; we can still
>> change
>> > the recommendation and skip 2.8.x as bridge release of course, if we
>> > think it would be better for users.
>> >
>> >
>> > Curious the get your feedback.
>> >
>> >
>> > -Matthias
>> >
>> >
>> > On 2/28/25 8:15 AM, Kuan Po Tseng wrote:
>> > > Hi Matthias,
>> > >
>> > > Feel free to edit the KIP directly – I'm totally fine with it. Thanks
>> a lot!
>> > >
>> > > Best,
>> > > Kuan-Po Tseng
>> > >
>> > > On 2025/02/28 07:35:10 "Matthias J. Sax" wrote:
>> > >> Thanks all. I am happy that we are making progress to close this out.
>> > >>
>> > >> As mentioned in my (long) email last Friday, I also think that the
>> KIP
>> > >> makes a lot of implicit assumptions, and it might benefit from begin
>> > >> more explicit, even explaining things that are "set in stone"
>> already,
>> > >> and the KIP does not change them (like API deprecation timelines and
>> > >> guarantees). But as Bruno said, it would help a lot to make it
>> easier to
>> > >> understand the KIP.
>> > >>
>> > >> Only a very small number of people seems to fully understand all the
>> > >> nuances, and it took quite a while to collect and curate all the
>> cases,
>> > >> to shape this KIP... And we know that a lot of people actually read
>> > >> KIPs, and the title for KIP-1124 is or sure click-bait...
>> > >>
>> > >> I am also happy to edit the wiki page directly if that's ok with
>> Kuan,
>> > >> to add background information that I believe is missing and
>> important.
>> > >>
>> > >>
>> > >>
>> > >> My overall understanding of the state of the KIP, ie, its actual
>> content
>> > >> seems to be, that there is a high level agreement, and the content of
>> > >> the KIP is ok.
>> > >>
>> > >> The only open question seems to be, if we want to recommend `2.8.x`
>> as a
>> > >> bridge release for Kafka Streams or not. As said in my email on
>> Friday,
>> > >> I am personally ok either way, with a preference to add `2.8.x` as
>> > >> bridge release, as it simplifies updating the code by avoiding
>> > >> compilation errors. With a 2.8.x bridge release, KS application code
>> > >> would _always_ compile, and the JavaDocs help a lot to guide users
>> how
>> > >> to migrate off deprecated APIs. -- With a direct upgrade from older
>> > >> versions to 3.9.x, code does not compile, and there is no guidance as
>> > >> the old methods (and corresponding JavaDocs) are already removed,
>> making
>> > >> it significant harder for user to update their code and make it
>> compile
>> > >> again.
>> > >>
>> > >> In the end, we would only _recommend_ 2.8.x as a bridge release to
>> > >> users, and there won't be any claim that an direct upgrade to 3.9.x
>>

[jira] [Created] (KAFKA-18907) add suitable error message when the appended value is too larger

2025-03-01 Thread Chia-Ping Tsai (Jira)
Chia-Ping Tsai created KAFKA-18907:
--

 Summary: add suitable error message when the appended value is too 
larger
 Key: KAFKA-18907
 URL: https://issues.apache.org/jira/browse/KAFKA-18907
 Project: Kafka
  Issue Type: Improvement
Reporter: Chia-Ping Tsai
Assignee: Chia-Ping Tsai


In ZooKeeper mode, large config values can be created by appending. In 
contrast, KRaft mode does not allow this. The root cause is that 
{{ConfigRecord}} assumes a maximum value size of {{{}Short.MAX_VALUE{}}}. This 
causes a runtime error, which is then converted to "UnknownServerException" for 
users, making the issue unclear.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


Re: [DISCUSS] KIP-1136: Make ConsumerGroupMetadata an interface

2025-03-01 Thread Paweł Szymczyk
Hi Kirk
Many thanks for very good questions!

KT1. I don't think that internal implementation should totally prevent of
passing alternative implementation, but for sure we need to log warning
when producer receives different than default implementaion. I think that
checking internally is the class implementation exactly
DefaultConsumerGroupMetadata  is against some basic principles like liskov
substition and if we like to design this mechanism that way, we shouldn't
expose an interface.
KT2. I am also not 100% satisfied with that name, I don't think that there
will be need for future different implementations as it is simple data
struct.
KT3. It will be place in producer/internals package and default java scope
for constructor and whole class.

@Matthias J. Sax  having those KT1 and KT2 in mind,
should we move forward with an interface or maybe ConsumerGroupMetadata has
to be a class, but we have to  limit possibility to create an instance
manualy by making the class final and changing constructor visibility to
private or package scope?

czw., 27 lut 2025 o 01:51 Kirk True  napisał(a):

> Hi Paweł,
>
> Thanks for the KIP!
>
> My questions:
>
> KT1. What will prevent developers from implementing their own
> ConsumerGroupMetadata and passing that to sendOffsetsToTransaction()? I
> assume the code will check the incoming object is of type
> DefaultConsumerGroupMetadata?
>
> KT2. To me, the use of the adjective "default" in
> DefaultConsumerGroupMetadata implies that there could be other
> implementations. Is the intention that there could be other implementations
> in the future?
>
> KT3. DefaultConsumerGroupMetadata should be defined in an "internals"
> package of some sort, right? Will users ever reference the implementation
> class name in their code? I'm assuming not.
>
> Thanks!
> Kirk
>
> On Tue, Feb 25, 2025, at 8:24 AM, Paweł Szymczyk wrote:
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1136%3A+Make+ConsumerGroupMetadata+an+interface
> >
> > --
> > Pozdrawiam
> > Paweł Szymczyk
> >
>


-- 
Pozdrawiam
Paweł Szymczyk