[jira] [Resolved] (KAFKA-17764) Remove unnecessary ignorable annotations from RPC schemas

[Chia-Ping Tsai (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-17764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chia-Ping Tsai resolved KAFKA-17764.

Resolution: Fixed

> Remove unnecessary ignorable annotations from RPC schemas
> -
>
> Key: KAFKA-17764
> URL: https://issues.apache.org/jira/browse/KAFKA-17764
> Project: Kafka
>  Issue Type: Sub-task
>Affects Versions: 4.0.0
>Reporter: Andrew Schofield
>Assignee: Andrew Schofield
>Priority: Major
> Fix For: 4.0.0
>
>
> ShareFetchRequest for example uses ignorable when it should not. It's 
> harmless but doesn't make sense.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-17772) Remove inControlledShutdownBrokers(Set) and unfenceBrokers(Set) from ReplicationControlManagerTest

[Chia-Ping Tsai (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-17772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chia-Ping Tsai resolved KAFKA-17772.

Fix Version/s: 4.0.0
   Resolution: Fixed

> Remove inControlledShutdownBrokers(Set) and 
> unfenceBrokers(Set) from ReplicationControlManagerTest
> 
>
> Key: KAFKA-17772
> URL: https://issues.apache.org/jira/browse/KAFKA-17772
> Project: Kafka
>  Issue Type: Test
>Reporter: Chia-Ping Tsai
>Assignee: Chia-Chuan Yu
>Priority: Major
> Fix For: 4.0.0
>
>
> The order of records must match the input order, so we should avoid using Set 
> as the input type since its order is undefined.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] KIP-1092: Extend Consumer#close with an option to leave the group or not

[Apoorv Mittal]

Hi TengYao,
Thanks for the KIP.

+1 (non-binding)

Regards,
Apoorv Mittal


On Fri, Oct 11, 2024 at 12:55 AM Sophie Blee-Goldman 
wrote:

> +1 (binding)
>
> Thanks for this KIP!
>
> On Mon, Oct 7, 2024 at 9:22 AM Kirk True  wrote:
>
> > Hi TengYao,
> >
> > +1 (non-binding)
> >
> > Thanks for all the work so far on this.
> >
> > Kirk
> >
> > On Mon, Oct 7, 2024, at 4:09 AM, TengYao Chi wrote:
> > > Hi Andrew,
> > >
> > > Thanks for reviewing and participating in the vote.
> > > I have corrected the issue as you pointed out.
> > >
> > > Sincerely,
> > > TengYao
> > >
> > > Andrew Schofield  於 2024年10月7日 週一
> > > 下午6:44寫道：
> > >
> > > > Thanks for the KIP.
> > > >
> > > > +1 (non-binding)
> > > >
> > > > I have one tiny nit that in the text but not the code snippet, you
> > mention
> > > > methods called
> > > > CloseOption.withTimeoutFluent(Duration timeout) and
> > > >
> CloseOption.withGroupMembershipOperationFluent(GroupMembershipOperation
> > > > operation).
> > > > I expect you meant to remove "Fluent" in all cases so the text
> matches
> > the
> > > > code snippet.
> > > >
> > > > Andrew
> > > >
> > > > 
> > > > From: TengYao Chi 
> > > > Sent: 07 October 2024 08:44
> > > > To: dev@kafka.apache.org 
> > > > Subject: Re: [VOTE] KIP-1092: Extend Consumer#close with an option to
> > > > leave the group or not
> > > >
> > > > Hi Chia-Ping,
> > > >
> > > > Thanks for pointing that out.
> > > > I originally wrote the full version to show the equivalent semantic
> > > > example, but you're absolutely right—it can be simplified by omitting
> > > > `.withGroupMembershipOperation(GroupMembershipOperation.DEFAULT)`
> since
> > > > it's the default value.
> > > >
> > > > This actually gave me an idea to include the shorter version for
> > clarity in
> > > > the explanation.
> > > > I will update it accordingly.
> > > >
> > > > Sincerely,
> > > > TengYao
> > > >
> > > > Chia-Ping Tsai  於 2024年10月7日 週一 下午1:13寫道：
> > > >
> > > > > +1 (binding)
> > > > >
> > > > > nit: in the "Migration Plan"
> > > > >
> > > > > ```
> > > > > consumer.close(CloseOption.timeout(Duration.ofSeconds(30))
> > > > >
>  .withGroupMembershipOperation(GroupMembershipOperation.DEFAULT));
> > > > > ```
> > > > >
> > > > > The sample above can likely be simplified, right?
> > > > >
> > > > > ```
> > > > > consumer.close(CloseOption.timeout(Duration.ofSeconds(30)));
> > > > > ```
> > > > >
> > > > > Best,
> > > > > Chia-Ping
> > > > >
> > > > > TengYao Chi  於 2024年10月7日 週一 上午10:23寫道：
> > > > >
> > > > > > Hi everyone,
> > > > > >
> > > > > > Based on our discussion
> > > > > > <
> https://lists.apache.org/thread/023mo7lk1vfvljjoovwbzwmw9wvf5t6m>
> > > > > >  regarding KIP-1092 <
> https://cwiki.apache.org/confluence/x/JQstEw>,
> > I
> > > > > > believe this KIP is now ready for a vote.
> > > > > >
> > > > > > Sincerely,
> > > > > > TengYao
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: New release branch 3.9

[Josep Prat]

Hi Colin,
I agree with you, this is not a blocker.
We merged those to trunk and 3.8 already, but without your feedback. I'll
try to address it today.

Best,
--
Josep Prat
Open Source Engineering Director, Aiven
josep.p...@aiven.io   |   +491715557497 | aiven.io
Aiven Deutschland GmbH
Alexanderufer 3-7, 10117 Berlin
Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen,
Anna Richardson, Kenneth Chen
Amtsgericht Charlottenburg, HRB 209739 B

On Thu, Oct 10, 2024, 23:43 Colin McCabe  wrote:

> Thanks for the PR, Josep.
>
> I don't think it's critical to re-generate the RC since we can simply
> update the docs on the website later as we've done in the past. If there is
> a new RC after RC2, it will be there, though.
>
> Also it looks like this will be a good fix for 3.8.1 as well :)
>
> best,
> Colin
>
> On Thu, Oct 10, 2024, at 09:51, Josep Prat wrote:
> > Hi Colin, if you haven't started probably it makes sense to merge
> > https://github.com/apache/kafka/pull/17447 as well.
> >
> >
> > Best,
> > --
> > Josep Prat
> > Open Source Engineering Director, Aiven
> > josep.p...@aiven.io   |   +491715557497 | aiven.io
> > Aiven Deutschland GmbH
> > Alexanderufer 3-7, 10117 Berlin
> > Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen,
> > Anna Richardson, Kenneth Chen
> > Amtsgericht Charlottenburg, HRB 209739 B
> >
> > On Thu, Oct 10, 2024, 18:31 Colin McCabe  wrote:
> >
> >> Hi all,
> >>
> >> I have merged KAFKA-17731 and KAFKA-17749 to 3.9. Also KAFKA-17751.
> >>
> >> We should be ready for the next RC now.
> >>
> >> Colin
> >>
> >>
> >> On Thu, Oct 10, 2024, at 08:45, Mickael Maison wrote:
> >> > Hi,
> >> >
> >> > Following up on https://issues.apache.org/jira/browse/KAFKA-17749. We
> >> > merged the fix in trunk and 3.8.
> >> > I opened a PR for 3.9: https://github.com/apache/kafka/pull/17457
> >> >
> >> > Thanks,
> >> > Mickael
> >> >
> >> > On Thu, Oct 10, 2024 at 3:14 PM Lianet M.  wrote:
> >> >>
> >> >> Hi Colin,
> >> >>
> >> >> Also, we discovered a bug affecting 3.9, that I would see as a
> >> regression
> >> >> given that it degrades the consumer#close.
> >> >>
> >> >> Fix has been merged to trunk with
> >> https://github.com/apache/kafka/pull/17431
> >> >> so wonder if we should include it.
> >> >>
> >> >> Thanks,
> >> >>
> >> >> Lianet
> >> >>
> >> >> On Thu, Oct 10, 2024 at 9:09 AM Mickael Maison <
> >> mickael.mai...@gmail.com>
> >> >> wrote:
> >> >>
> >> >> > Hi Colin,
> >> >> >
> >> >> > In Kafka 3.8 we accidentally renamed a metric:
> >> >> > https://issues.apache.org/jira/browse/KAFKA-17749
> >> >> > I wonder if we should include the fix to revert it to its original
> >> >> > name in 3.9.0 (and 3.8.1) to limit the impact:
> >> >> > https://github.com/apache/kafka/pull/17430
> >> >> >
> >> >> > Thanks,
> >> >> > Mickael
> >> >> >
> >> >> > On Wed, Oct 9, 2024 at 6:38 PM Josep Prat
>  >> >
> >> >> > wrote:
> >> >> > >
> >> >> > > Thanks Colin,
> >> >> > >
> >> >> > > I reviewed the PR you shared.
> >> >> > >
> >> >> > > Best,
> >> >> > >
> >> >> > > On Wed, Oct 9, 2024 at 6:26 PM Colin McCabe 
> >> wrote:
> >> >> > >
> >> >> > > > Hi Josep,
> >> >> > > >
> >> >> > > > I marked this as a blocker. We will take a look. I also found
> some
> >> >> > > > dependencies that need to be updated due to CVEs. See
> >> >> > > > github.com/apache/kafka/pull/17436
> >> >> > > >
> >> >> > > > best,
> >> >> > > > Colin
> >> >> > > >
> >> >> > > > On Wed, Oct 9, 2024, at 05:52, Josep Prat wrote:
> >> >> > > > > Hi Colin,
> >> >> > > > >
> >> >> > > > > I want to raise a bug we encountered while testing 3.9 RC0.
> You
> >> can
> >> >> > find
> >> >> > > > > the report here:
> >> https://issues.apache.org/jira/browse/KAFKA-17751
> >> >> > > > > Its current severity is "high", but IMO it might even be a
> >> blocker.
> >> >> > What
> >> >> > > > do
> >> >> > > > > you think?
> >> >> > > > >
> >> >> > > > > Best,
> >> >> > > > >
> >> >> > > > > On Fri, Oct 4, 2024 at 5:18 PM José Armando García Sancio
> >> >> > > > >  wrote:
> >> >> > > > >
> >> >> > > > >> Thanks Colin.
> >> >> > > > >>
> >> >> > > > >> KAFKA-16927 has been merged to trunk and the 3.9 branch.
> >> >> > > > >>
> >> >> > > > >> --
> >> >> > > > >> -José
> >> >> > > > >>
> >> >> > > > >
> >> >> > > > >
> >> >> > > > > --
> >> >> > > > > [image: Aiven]
> <
> https://www.aiven.io>
> >> >> > > > >
> >> >> > > > > *Josep Prat*
> >> >> > > > > Open Source Engineering Director, *Aiven*
> >> >> > > > > josep.p...@aiven.io   |   +491715557497
> >> >> > > > > aiven.io    |   <
> >> >> > > > https://www.facebook.com/aivencloud>
> >> >> > > > >      <
> >> >> > > > https://twitter.com/aiven_io>
> >> >> > > > > *Aiven Deutschland GmbH*
> >> >> > > > > Alexanderufer 3-7, 10117 Berlin
> >> >> > > > > Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen,
> >> >> > > > > Anna Richardson, Kenneth Chen
> >> >> > > >

[jira] [Resolved] (KAFKA-17775) Remove Java#IS_JAVA9_COMPATIBLE property

[Jira]

 [ 
https://issues.apache.org/jira/browse/KAFKA-17775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

黃竣陽 resolved KAFKA-17775.
-
Resolution: Duplicate

> Remove Java#IS_JAVA9_COMPATIBLE property
> 
>
> Key: KAFKA-17775
> URL: https://issues.apache.org/jira/browse/KAFKA-17775
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: 黃竣陽
>Assignee: 黃竣陽
>Priority: Major
> Fix For: 4.0.0
>
>
> Because the Java 8 is deprecate in Kafka, so we didn't need this check



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-17779) Fix flaky RemoteLogManagerTest#testFetchOffsetByTimestampWithTieredStorageDoesNotFetchIndexWhenExistsLocally

[Chia-Ping Tsai (Jira)]

Chia-Ping Tsai created KAFKA-17779:
--

 Summary: Fix flaky 
RemoteLogManagerTest#testFetchOffsetByTimestampWithTieredStorageDoesNotFetchIndexWhenExistsLocally
 Key: KAFKA-17779
 URL: https://issues.apache.org/jira/browse/KAFKA-17779
 Project: Kafka
  Issue Type: Test
Reporter: Chia-Ping Tsai
Assignee: Chia-Ping Tsai


```
org.mockito.exceptions.misusing.WrongTypeOfReturnValue:
SingletonList cannot be returned by leaderEpochCache()
leaderEpochCache() should return Option
***
If you're unsure why you're getting above error read on.
Due to the nature of the syntax above problem might occur because:
1. This exception might occur in wrongly written multi-threaded tests.
Please refer to Mockito FAQ on limitations of concurrency testing.
2. A spy is stubbed using when(spy.foo()).then() syntax. It is safer to stub 
spies -
- with doReturn
```

https://github.com/apache/kafka/actions/runs/11295770598/attempts/1#summary-31419257288



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-17778) Clean up client instance cache on connection close

[Apoorv Mittal (Jira)]

Apoorv Mittal created KAFKA-17778:
-

 Summary: Clean up client instance cache on connection close
 Key: KAFKA-17778
 URL: https://issues.apache.org/jira/browse/KAFKA-17778
 Project: Kafka
  Issue Type: Improvement
Reporter: Apoorv Mittal
Assignee: Apoorv Mittal


As with https://issues.apache.org/jira/browse/KAFKA-17776, there is a way to 
attach listener which can react on client connection disconnection. Hence clean 
up client instance cache when respective connections are disconnected.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Jenkins build became unstable: Kafka » Kafka PowerPC Daily » test-powerpc #84

[Apache Jenkins Server]

See 

[jira] [Resolved] (KAFKA-17199) add unit tests for TransactionLogConfig and TransactionStateManagerConfig

[Chia-Ping Tsai (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-17199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chia-Ping Tsai resolved KAFKA-17199.

Fix Version/s: 4.0.0
   Resolution: Fixed

> add unit tests for TransactionLogConfig and TransactionStateManagerConfig
> -
>
> Key: KAFKA-17199
> URL: https://issues.apache.org/jira/browse/KAFKA-17199
> Project: Kafka
>  Issue Type: Test
>Reporter: Chia-Ping Tsai
>Assignee: Ming-Yen Chung
>Priority: Minor
> Fix For: 4.0.0
>
>
> This is a follow-up of https://issues.apache.org/jira/browse/KAFKA-17195
> `transaction-coordinator` module has no tests and that is prone to cause 
> build failure (since we normally assume each module has tests). Hence, adding 
> tests not only increase test coverage but also fix possible bugs.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-1096: Add ability to pass headers to ProducerPerformance.

[Andrew Schofield]

Hi Max,
Thanks for the KIP.

I think this is a useful addition to the ProducerPerformance tool. I think the 
KIP needs some more detail.

If you're adding `--headers`, you should specify how to give multiple headers. 
For example, is it
`--headers K1=V1,K2=V2` or do you have to do `--headers K1=V1 --headers K2=V2`?

If you use `--headers-file`, what is the format of the file?

And I'd like to suggest that you also extend the tool with a 
`--bootstrap-server` option. It's annoying when
using `kafka-producer-perf-test.sh` and `kafka-consumer-perf-test.sh` together 
that they are inconsistent in
how you provide the bootstrap server address. Feel free to ignore this 
suggestion, but since you're in there

Thanks,
Andrew


From: Maxim Fortun 
Sent: 11 October 2024 02:10
To: dev@kafka.apache.org 
Subject: [DISCUSS] KIP-1096: Add ability to pass headers to ProducerPerformance.

Hi all,
I would like to introduce a minor enhancement to ProducerPerformance class to 
pass headers to the create produce requests.
KIP: https://cwiki.apache.org/confluence/display/KAFKA/KIP-1096

Headers are useful for tracking many aspects of the performance testing. Like 
regions, hosts, etc...
Headers passed downstream to consumers can also drive perf test behavior.

I have submitted a PR(https://github.com/apache/kafka/pull/17462)  for to add 
--headers arg, only to find a bit later that there is a similar 
PR(https://github.com/apache/kafka/pull/9254) from 2020 that adds 
--headers-file arg.

Looks like there was no traction on that since 2020.
I did not create a new JIRA issue, I added a comment  with the second PR to an 
existing issue(https://issues.apache.org/jira/browse/KAFKA-10462) from 2020.
We can add either one, or both features.

Please take a look.
Any and all feedback is greatly appreciated.
Thanks,
Max

[jira] [Created] (KAFKA-17773) Upgrade spotbug to work under java 23

[Chia-Ping Tsai (Jira)]

Chia-Ping Tsai created KAFKA-17773:
--

 Summary: Upgrade spotbug to work under java 23
 Key: KAFKA-17773
 URL: https://issues.apache.org/jira/browse/KAFKA-17773
 Project: Kafka
  Issue Type: Sub-task
Reporter: Chia-Ping Tsai
Assignee: Chia-Chuan Yu


It seems `6.0.24` and `4.8.6` work well



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-17777) Flaky KafkaConsumerTest testReturnRecordsDuringRebalance

[Lianet Magrans (Jira)]

Lianet Magrans created KAFKA-1:
--

 Summary: Flaky KafkaConsumerTest testReturnRecordsDuringRebalance
 Key: KAFKA-1
 URL: https://issues.apache.org/jira/browse/KAFKA-1
 Project: Kafka
  Issue Type: Test
  Components: clients, unit tests
Reporter: Lianet Magrans


Top flaky consumer test at the moment (after several recent fixes addressing 
consumer tests flakiness): 
[https://ge.apache.org/scans/tests?search.rootProjectNames=kafka&search.tags=trunk&search.timeZoneId=America%2FNew_York&tests.sortField=FLAKY]
 


It's been flaky in trunk for a while:

[https://ge.apache.org/scans/tests?search.relativeStartTime=P28D&search.rootProjectNames=kafka&search.tags=trunk&search.timeZoneId=America%2FToronto&tests.container=org.apache.kafka.clients.consumer.KafkaConsumerTest&tests.sortField=FLAKY&tests.test=testReturnRecordsDuringRebalance(GroupProtocol)%5B1%5D]
 

 

Fails with : org.opentest4j.AssertionFailedError: Condition not met within 
timeout 15000. Does not complete rebalance in time ==> expected:  but 
was: 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-17774) Add capability for max fetch records in share fetch

[Apoorv Mittal (Jira)]

Apoorv Mittal created KAFKA-17774:
-

 Summary: Add capability for max fetch records in share fetch
 Key: KAFKA-17774
 URL: https://issues.apache.org/jira/browse/KAFKA-17774
 Project: Kafka
  Issue Type: Sub-task
Reporter: Apoorv Mittal
Assignee: Apoorv Mittal






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-17717) Remove ConfigUtils#translateDeprecatedConfigs

[Chia-Ping Tsai (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-17717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chia-Ping Tsai resolved KAFKA-17717.

Fix Version/s: 4.0.0
   Resolution: Fixed

> Remove ConfigUtils#translateDeprecatedConfigs
> -
>
> Key: KAFKA-17717
> URL: https://issues.apache.org/jira/browse/KAFKA-17717
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Chia-Ping Tsai
>Assignee: Mingdao Yang
>Priority: Minor
> Fix For: 4.0.0
>
>
> They are no longer needed. We can always retrieve them via Git if new 
> deprecated configs are introduced in the future.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-17661) Fix flaky BufferPoolTest.testBlockTimeout

[Chia-Ping Tsai (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-17661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chia-Ping Tsai resolved KAFKA-17661.

Fix Version/s: 4.0.0
   Resolution: Fixed

> Fix flaky BufferPoolTest.testBlockTimeout
> -
>
> Key: KAFKA-17661
> URL: https://issues.apache.org/jira/browse/KAFKA-17661
> Project: Kafka
>  Issue Type: Bug
>  Components: clients, unit tests
>Reporter: Yu-Lin Chen
>Assignee: Yu-Lin Chen
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: 
> 0001-reproduce-racing-issue-by-adding-delay-to-test-thread.patch
>
>
> 4 flaky out of 221 trunk build in the past 28 days. (github) ([Report 
> Link|https://ge.apache.org/scans/tests?search.rootProjectNames=kafka&search.startTimeMax=1727681219558&search.startTimeMin=172520640&search.tags=github&search.timeZoneId=Asia%2FTaipei&tests.container=org.apache.kafka.clients.producer.internals.BufferPoolTest&tests.test=testBlockTimeout()])
> ([Sep 27 2024 at 
> 04:54:00|https://ge.apache.org/s/nh44u7tsm2lri/tests/task/:clients:test/details/org.apache.kafka.clients.producer.internals.BufferPoolTest/testBlockTimeout()?expanded-stacktrace=WyIwIl0&top-execution=1])
> {code:java}
> org.opentest4j.AssertionFailedError: The buffer allocated more memory than 
> its maximum value 10   
> at org.junit.jupiter.api.AssertionUtils.fail(AssertionUtils.java:38)  
> at org.junit.jupiter.api.Assertions.fail(Assertions.java:138) 
> at 
> org.apache.kafka.clients.producer.internals.BufferPoolTest.testBlockTimeout(BufferPoolTest.java:184)
>
> at java.lang.reflect.Method.invoke(Method.java:566)   
> at java.util.ArrayList.forEach(ArrayList.java:1541)   
> at java.util.ArrayList.forEach(ArrayList.java:1541)
> {code}
> Root cause:
>  # The test relies on 3 asynchronous threads being triggered in parallel with 
> the test thread [1]. However, there is no guarantee of parallelism in test 
> environment. The issue will happend if test thread didn't get CPU within 25 
> ms. We could reproduce this issue by adding 30 ms delay to test thread. 
> Please check the attached patch.
>  # Since a 25 ms delay is obviously unreliable in the test environment, we 
> could consider rewriting the test or increasing the delay. (The 
> maxBlockTimeMs was reduced from 2000ms to 10 ms in KAFKA-9852)
> [1] 
> [https://github.com/apache/kafka/blob/40360819bb97d6b05dfef6451888b4d908fc3bf4/clients/src/test/java/org/apache/kafka/clients/producer/internals/BufferPoolTest.java#L175-L179]



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Request for re-enablement of Jenkins CI

[Prashant Jagtap]

Hi Team,

I hope this email finds you well.

I am writing to kindly request the re-enablement of Jenkins CI for s390x 
architecture. We understand that resource and maintenance efforts are involved, 
and we greatly appreciate your ongoing support. If there are any specific 
requirements or steps we need to follow to assist with this process, please let 
us know—we’re more than happy to collaborate to make this happen.
Looking forward to your positive response. Thank you for your consideration.


Thanks and Regards
Prashant Jagtap

[jira] [Created] (KAFKA-17776) Add connection disconnect listener in socket server

[Apoorv Mittal (Jira)]

Apoorv Mittal created KAFKA-17776:
-

 Summary: Add connection disconnect listener in socket server
 Key: KAFKA-17776
 URL: https://issues.apache.org/jira/browse/KAFKA-17776
 Project: Kafka
  Issue Type: Improvement
Reporter: Apoorv Mittal
Assignee: Apoorv Mittal


There is currently no way to act or cleanup resources when connections are 
dropped from Kafka Broker. The socket server knows about the disconnected 
connections and has a way to process disconnected KafkaChannel/connections but 
there isn't any additional listeners which can be attached to act on connection 
disconnections.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-17775) Remove Java#IS_JAVA9_COMPATIBLE property

[Jira]

黃竣陽 created KAFKA-17775:
---

 Summary: Remove Java#IS_JAVA9_COMPATIBLE property
 Key: KAFKA-17775
 URL: https://issues.apache.org/jira/browse/KAFKA-17775
 Project: Kafka
  Issue Type: Improvement
Reporter: 黃竣陽
Assignee: 黃竣陽
 Fix For: 4.0.0


Because the Java 8 is deprecate in Kafka, so we didn't need this check



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-17780) Heartbeat interval is not configured in the heartbeatrequest manager

[Arpit Goyal (Jira)]

Arpit Goyal created KAFKA-17780:
---

 Summary: Heartbeat interval is not configured in the 
heartbeatrequest manager
 Key: KAFKA-17780
 URL: https://issues.apache.org/jira/browse/KAFKA-17780
 Project: Kafka
  Issue Type: Bug
  Components: consumer
Reporter: Arpit Goyal


In the AbstractHeartBeatRequestManager  , I observed we are not setting the 
right heartbeat request interval. Is this intentional ?
[~lianetm] [~kirktrue]
{code:java}
long retryBackoffMs = config.getLong(ConsumerConfig.RETRY_BACKOFF_MS_CONFIG);
long retryBackoffMaxMs = 
config.getLong(ConsumerConfig.RETRY_BACKOFF_MAX_MS_CONFIG);
this.heartbeatRequestState = new HeartbeatRequestState(logContext, 
time, 0, retryBackoffMs,
retryBackoffMaxMs, maxPollIntervalMs);
{code}




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-17723) Fix "this-escape" compiler warnings (MultiThreadedEventProcessor and DistributedHerder) for JDK 23

[Chia-Ping Tsai (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-17723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chia-Ping Tsai resolved KAFKA-17723.

Fix Version/s: 4.0.0
   Resolution: Fixed

> Fix "this-escape" compiler warnings (MultiThreadedEventProcessor and 
> DistributedHerder) for JDK 23
> --
>
> Key: KAFKA-17723
> URL: https://issues.apache.org/jira/browse/KAFKA-17723
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Chia-Ping Tsai
>Assignee: Anshul Goyal
>Priority: Minor
> Fix For: 4.0.0
>
>
> {code:java}
> > Task :coordinator-common:compileJava FAILED
> /root/kafka/coordinator-common/src/main/java/org/apache/kafka/coordinator/common/runtime/MultiThreadedEventProcessor.java:107:
>  warning: [this-escape] possible 'this' escape before subclass is fully 
> initialized
>         this.threads = IntStream.range(0, numThreads).mapToObj(threadId ->
>                                                                ^
> error: warnings found and -Werror specified
>  {code}
> {code:java}
> > Task :connect:runtime:compileJava
> /root/kafka/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/distributed/DistributedHerder.java:269:
>  warning: [this-escape] possible 'this' escape before subclass is fully 
> initialized
>         this(config, worker, worker.workerId(), kafkaClusterId, 
> statusBackingStore, configBackingStore, null, restUrl, restClient, 
> worker.metrics(),
>             ^
> /root/kafka/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/distributed/DistributedHerder.java:318:
>  warning: [this-escape] previous possible 'this' escape happens here via 
> invocation
>                               new RebalanceListener(time), time, clientId, 
> logContext);
>  {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-1058: Txn consumer exerts pressure on remote storage when reading non-txn topic

[Kamal Chandraprakash]

Bump for review.

If the additional proposal looks good, I'll append them to the KIP. PTAL.

New API in RLMM#nextRemoteLogSegmentMetadataWithTxnIndex

--
Kamal

On Sun, Oct 6, 2024 at 7:20 PM Kamal Chandraprakash <
kamal.chandraprak...@gmail.com> wrote:

> Hi Christo,
>
> Thanks for the review!
>
> Adding the new API `nextRemoteLogSegmentMetadataWithTxnIndex` in RLMM
> helps to
> reduce the complexity of linear search. With this API, we have to:
>
> 1. Maintain one more skip-list [1] for each of the epochs in the partition
> in RLMM that might
> increase the memory usage of TopicBased RLMM implementation.
> 1a) The skip-list will be empty when there are no aborted txn entries
> for a partition/epoch which is the predominant case.
> 1b) The skip-list will act as a duplicate when *most* of the segments
> have aborted txn entries, assuming aborted txn are quite low, this should
> be fine.
> 2. Change the logic to retrieve the aborted txns (we have to query the
> nextRLSMWithTxnIndex
> for each of the leader-epoch).
> 3. Logic divergence from how we retrieve the aborted txn entries compared
> to the local-log.
>
> The approach looks good to me. If everyone is aligned, then we can proceed
> to add this API to RLMM.
>
> Another option I was thinking of is to capture the `lastStableOffsetLag`
> [2] while rotating the segment.
> But, that is a bigger change we can take later.
>
> [1]:
> https://sourcegraph.com/github.com/apache/kafka/-/blob/storage/src/main/java/org/apache/kafka/server/log/remote/metadata/storage/RemoteLogLeaderEpochState.java?L43
> [2]:
> https://sourcegraph.com/github.com/apache/kafka/-/blob/core/src/main/scala/kafka/log/UnifiedLog.scala?L432
>
>
> Thanks,
> Kamal
>
> On Fri, Oct 4, 2024 at 4:21 PM Christo Lolov 
> wrote:
>
>> Heya,
>>
>> Apologies for the delay. I have been thinking about this problem recently
>> as well and while I believe storing a boolean in the metadata is good, I
>> think we can do better by introducing a new method to the RLMM along the
>> lines of
>>
>> Optional
>> nextRemoteLogSegmentMetadataWithTxnIndex(TopicIdPartition
>> topicIdPartition,
>> int epochForOffset, long offset) throws RemoteStorageException
>>
>> This will help plugin implementers to build optimisations such as skip
>> lists which will give them the next segment quicker than a linear search.
>>
>> I am keen to hear your thoughts!
>>
>> Best,
>> Christo
>>
>> On Fri, 4 Oct 2024 at 10:48, Kamal Chandraprakash <
>> kamal.chandraprak...@gmail.com> wrote:
>>
>> > Hi Luke,
>> >
>> > Thanks for the review!
>> >
>> > > Do you think it is helpful if we store the "least abort start offset
>> in
>> > the
>> > segment", and -1 means no txnIndex. So that we can have a way to know
>> if we
>> > need to fetch this txn index or not.
>> >
>> > 1. No, this change won't have an effect. To find the upper-bound offset
>> > [1], we have to
>> > fetch that segment's offset index file. The RemoteIndexCache [2]
>> > fetches all the 3
>> > index files together and caches them for subsequent use, so this
>> > improvement
>> > won't have an effect as the current segment txn index gets
>> downloaded
>> > anyway.
>> >
>> > 2. The reason for choosing boolean is to make the change backward
>> > compatible.
>> >  There can be existing RLM events for the uploaded segments. The
>> > default
>> >  value of `txnIdxEmpty` is false so the *old* RLM events are
>> assumed to
>> > contain
>> >  the txn index files and those files are downloaded if they exist.
>> >
>> > [1]:
>> >
>> >
>> https://sourcegraph.com/github.com/apache/kafka@trunk/-/blob/core/src/main/java/kafka/log/remote/RemoteLogManager.java?L1732
>> > [2]:
>> >
>> >
>> https://sourcegraph.com/github.com/apache/kafka@trunk/-/blob/storage/src/main/java/org/apache/kafka/storage/internals/log/RemoteIndexCache.java?L383
>> >
>> > Thanks,
>> > Kamal
>> >
>> > On Thu, Oct 3, 2024 at 3:11 PM Luke Chen  wrote:
>> >
>> > > Hi Kamal,
>> > >
>> > > Sorry for the late review.
>> > > Thanks for the KIP, this will improve the transaction reading for
>> remote
>> > > storage.
>> > > Overall LGTM, just one minor thought:
>> > >
>> > > Currently, we only store the `TxnIndexEmpty` bool value in the segment
>> > > metadata.
>> > > Do you think it is helpful if we store the "least abort start offset
>> in
>> > the
>> > > segment", and -1 means no txnIndex. So that we can have a way to know
>> if
>> > we
>> > > need to fetch this txn index or not.
>> > >
>> > > Thanks.
>> > > Luke
>> > >
>> > > On Mon, Sep 9, 2024 at 3:26 PM Kamal Chandraprakash <
>> > > kamal.chandraprak...@gmail.com> wrote:
>> > >
>> > > > Hi all,
>> > > >
>> > > > If there are no more comments, I'll start a voting thread soon.
>> > > >
>> > > > Thanks,
>> > > > Kamal
>> > > >
>> > > > On Fri, Sep 6, 2024 at 7:28 PM Kamal Chandraprakash <
>> > > > kamal.chandraprak...@gmail.com> wrote:
>> > > >
>> > > > > Bumping this thread again for review!
>> > > > >
>> > > > > Reduced the scope of the proposa