[jira] [Resolved] (KAFKA-16272) Update connect_distributed_test.py to support KIP-848’s group protocol config

[Kirk True (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-16272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kirk True resolved KAFKA-16272.
---
Resolution: Fixed

> Update connect_distributed_test.py to support KIP-848’s group protocol config
> -
>
> Key: KAFKA-16272
> URL: https://issues.apache.org/jira/browse/KAFKA-16272
> Project: Kafka
>  Issue Type: Test
>  Components: clients, consumer, system tests
>Affects Versions: 3.7.0
>Reporter: Kirk True
>Assignee: Sagar Rao
>Priority: Major
>  Labels: kip-848-client-support, system-tests
> Fix For: 3.8.0
>
>
> This task is to update the test method(s) in {{connect_distributed_test.py}} 
> to support the {{group.protocol}} configuration introduced in 
> [KIP-848|https://cwiki.apache.org/confluence/display/KAFKA/KIP-848%3A+The+Next+Generation+of+the+Consumer+Rebalance+Protocol]
>  by adding an optional {{group_protocol}} argument to the tests and matrixes.
> See KAFKA-16231 as an example of how the test parameters can be changed.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-17603) KRaftMigrationDriverTest testNoDualWriteBeforeMigration became flaky

[PoAn Yang (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-17603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

PoAn Yang resolved KAFKA-17603.
---
Resolution: Not A Problem

The test case is removed in trunk. We can reopen again if it's flaky in 3.x.

> KRaftMigrationDriverTest testNoDualWriteBeforeMigration became flaky
> 
>
> Key: KAFKA-17603
> URL: https://issues.apache.org/jira/browse/KAFKA-17603
> Project: Kafka
>  Issue Type: Test
>  Components: kraft
>Reporter: David Arthur
>Assignee: PoAn Yang
>Priority: Minor
> Attachments: image-2024-09-24-13-25-22-534.png
>
>
> This test suddenly became consistently flaky on trunk on Sep 16. 
> !image-2024-09-24-13-25-22-534.png|width=725,height=285!
> [https://ge.apache.org/scans/tests?search.names=CI%20workflow&search.relativeStartTime=P28D&search.rootProjectNames=kafka&search.tags=github,trunk&search.tasks=test&search.timeZoneId=America%2FNew_York&search.values=CI&tests.container=org.apache.kafka.metadata.migration.KRaftMigrationDriverTest&tests.test=testNoDualWriteBeforeMigration()]
>  
> There haven't been any changes to this test or the metadata module in that 
> time frame, so this is pretty weird. The commit when this became flaky was 
> [https://github.com/apache/kafka/commit/d0f4d691b592ed2d65c9df413f5d6660df7fc90e.|https://github.com/apache/kafka/commit/d0f4d691b592ed2d65c9df413f5d6660df7fc90e]
>  
> Marking this a minor since this test will be deleted soon.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] KIP-1092: Extend Consumer#close with an option to leave the group or not

[Chia-Ping Tsai]

+1 (binding)

nit: in the "Migration Plan"

```
consumer.close(CloseOption.timeout(Duration.ofSeconds(30))
.withGroupMembershipOperation(GroupMembershipOperation.DEFAULT));
```

The sample above can likely be simplified, right?

```
consumer.close(CloseOption.timeout(Duration.ofSeconds(30)));
```

Best,
Chia-Ping

TengYao Chi  於 2024年10月7日 週一 上午10:23寫道：

> Hi everyone,
>
> Based on our discussion
> 
>  regarding KIP-1092 , I
> believe this KIP is now ready for a vote.
>
> Sincerely,
> TengYao
>

[jira] [Resolved] (KAFKA-17529) Remove blacklist from MM2

[Chia-Ping Tsai (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-17529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chia-Ping Tsai resolved KAFKA-17529.

Fix Version/s: 4.0.0
   Resolution: Fixed

> Remove blacklist from MM2
> -
>
> Key: KAFKA-17529
> URL: https://issues.apache.org/jira/browse/KAFKA-17529
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Chia-Ping Tsai
>Assignee: Chia-Chuan Yu
>Priority: Major
> Fix For: 4.0.0
>
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-17695) cleanup org.apache.kafka.common.test.TestUtils

[Chia-Ping Tsai (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-17695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chia-Ping Tsai resolved KAFKA-17695.

Fix Version/s: 4.0.0
   Resolution: Fixed

> cleanup org.apache.kafka.common.test.TestUtils
> --
>
> Key: KAFKA-17695
> URL: https://issues.apache.org/jira/browse/KAFKA-17695
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Chia-Ping Tsai
>Assignee: 黃竣陽
>Priority: Minor
> Fix For: 4.0.0
>
>
> It is copied from client module, so there are many unused code. Also, the 
> method should not be exposed publicly.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[VOTE] KIP-1092: Extend Consumer#close with an option to leave the group or not

[TengYao Chi]

Hi everyone,

Based on our discussion

 regarding KIP-1092 , I
believe this KIP is now ready for a vote.

Sincerely,
TengYao

[jira] [Created] (KAFKA-17707) Remove zk from BaseConsumerTest

[PoAn Yang (Jira)]

PoAn Yang created KAFKA-17707:
-

 Summary: Remove zk from BaseConsumerTest
 Key: KAFKA-17707
 URL: https://issues.apache.org/jira/browse/KAFKA-17707
 Project: Kafka
  Issue Type: Sub-task
Reporter: PoAn Yang
Assignee: PoAn Yang


Remove {{{}Arguments.of("zk", "classic"){}}}.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-12895) KIP-751: Drop support for Scala 2.12 in Kafka 4.0 (deprecate in 3.0)

[Chia-Ping Tsai (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-12895?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chia-Ping Tsai resolved KAFKA-12895.

Resolution: Fixed

> KIP-751: Drop support for Scala 2.12 in Kafka 4.0 (deprecate in 3.0)
> 
>
> Key: KAFKA-12895
> URL: https://issues.apache.org/jira/browse/KAFKA-12895
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Ismael Juma
>Assignee: TengYao Chi
>Priority: Major
>  Labels: kip
> Fix For: 4.0.0
>
>
> We propose to deprecate Scala 2.12 support n Apache Kafka 3.0 and to drop it 
> in Apache Kafka 4.0.
>  
> KIP: 
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=181308218



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-17709) remove retry_zinc

[Chia-Ping Tsai (Jira)]

Chia-Ping Tsai created KAFKA-17709:
--

 Summary: remove retry_zinc
 Key: KAFKA-17709
 URL: https://issues.apache.org/jira/browse/KAFKA-17709
 Project: Kafka
  Issue Type: Improvement
Reporter: Chia-Ping Tsai
Assignee: Chia-Ping Tsai


The Jenkins CI was removed in [this 
commit|https://github.com/apache/kafka/commit/264131cdaaef3f4696942f26534b3f61f3a2a162],
 so the 'retry_zinc' file is no longer in use.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-17415) Avoid overflow of expiried timestamp

[Chia-Ping Tsai (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-17415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chia-Ping Tsai resolved KAFKA-17415.

Resolution: Fixed

> Avoid overflow of expiried timestamp
> 
>
> Key: KAFKA-17415
> URL: https://issues.apache.org/jira/browse/KAFKA-17415
> Project: Kafka
>  Issue Type: Bug
>Reporter: Chia-Ping Tsai
>Assignee: TengYao Chi
>Priority: Major
> Fix For: 4.0.0
>
>
> Both of zk and kraft modes do not handle the overflow, and hence setting a 
> large max life time will cause a negative expired timestamp and negative max 
> timestamp. That is unexpected behavior.
> [0] 
> [https://github.com/apache/kafka/blob/0123bdcf06d2554ea62ba3a75af7c4b17eb4d23a/metadata/src/main/java/org/apache/kafka/controller/DelegationTokenControlManager.java#L191|https://github.com/apache/kafka/blob/4e846038a6313e28d3b3ee3d3c3163d1f26356bf/metadata/src/main/java/org/apache/kafka/controller/DelegationTokenControlManager.java#L188]
> [1] 
> [https://github.com/apache/kafka/blob/0123bdcf06d2554ea62ba3a75af7c4b17eb4d23a/core/src/main/scala/kafka/server/DelegationTokenManagerZk.scala#L182|https://github.com/apache/kafka/blob/0123bdcf06d2554ea62ba3a75af7c4b17eb4d23a/core/src/main/scala/kafka/server/DelegationTokenManagerZk.scala#L297]



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-1058: Txn consumer exerts pressure on remote storage when reading non-txn topic

[Kamal Chandraprakash]

Hi Christo,

Thanks for the review!

Adding the new API `nextRemoteLogSegmentMetadataWithTxnIndex` in RLMM helps
to
reduce the complexity of linear search. With this API, we have to:

1. Maintain one more skip-list [1] for each of the epochs in the partition
in RLMM that might
increase the memory usage of TopicBased RLMM implementation.
1a) The skip-list will be empty when there are no aborted txn entries
for a partition/epoch which is the predominant case.
1b) The skip-list will act as a duplicate when *most* of the segments
have aborted txn entries, assuming aborted txn are quite low, this should
be fine.
2. Change the logic to retrieve the aborted txns (we have to query the
nextRLSMWithTxnIndex
for each of the leader-epoch).
3. Logic divergence from how we retrieve the aborted txn entries compared
to the local-log.

The approach looks good to me. If everyone is aligned, then we can proceed
to add this API to RLMM.

Another option I was thinking of is to capture the `lastStableOffsetLag`
[2] while rotating the segment.
But, that is a bigger change we can take later.

[1]:
https://sourcegraph.com/github.com/apache/kafka/-/blob/storage/src/main/java/org/apache/kafka/server/log/remote/metadata/storage/RemoteLogLeaderEpochState.java?L43
[2]:
https://sourcegraph.com/github.com/apache/kafka/-/blob/core/src/main/scala/kafka/log/UnifiedLog.scala?L432


Thanks,
Kamal

On Fri, Oct 4, 2024 at 4:21 PM Christo Lolov  wrote:

> Heya,
>
> Apologies for the delay. I have been thinking about this problem recently
> as well and while I believe storing a boolean in the metadata is good, I
> think we can do better by introducing a new method to the RLMM along the
> lines of
>
> Optional
> nextRemoteLogSegmentMetadataWithTxnIndex(TopicIdPartition topicIdPartition,
> int epochForOffset, long offset) throws RemoteStorageException
>
> This will help plugin implementers to build optimisations such as skip
> lists which will give them the next segment quicker than a linear search.
>
> I am keen to hear your thoughts!
>
> Best,
> Christo
>
> On Fri, 4 Oct 2024 at 10:48, Kamal Chandraprakash <
> kamal.chandraprak...@gmail.com> wrote:
>
> > Hi Luke,
> >
> > Thanks for the review!
> >
> > > Do you think it is helpful if we store the "least abort start offset in
> > the
> > segment", and -1 means no txnIndex. So that we can have a way to know if
> we
> > need to fetch this txn index or not.
> >
> > 1. No, this change won't have an effect. To find the upper-bound offset
> > [1], we have to
> > fetch that segment's offset index file. The RemoteIndexCache [2]
> > fetches all the 3
> > index files together and caches them for subsequent use, so this
> > improvement
> > won't have an effect as the current segment txn index gets downloaded
> > anyway.
> >
> > 2. The reason for choosing boolean is to make the change backward
> > compatible.
> >  There can be existing RLM events for the uploaded segments. The
> > default
> >  value of `txnIdxEmpty` is false so the *old* RLM events are assumed
> to
> > contain
> >  the txn index files and those files are downloaded if they exist.
> >
> > [1]:
> >
> >
> https://sourcegraph.com/github.com/apache/kafka@trunk/-/blob/core/src/main/java/kafka/log/remote/RemoteLogManager.java?L1732
> > [2]:
> >
> >
> https://sourcegraph.com/github.com/apache/kafka@trunk/-/blob/storage/src/main/java/org/apache/kafka/storage/internals/log/RemoteIndexCache.java?L383
> >
> > Thanks,
> > Kamal
> >
> > On Thu, Oct 3, 2024 at 3:11 PM Luke Chen  wrote:
> >
> > > Hi Kamal,
> > >
> > > Sorry for the late review.
> > > Thanks for the KIP, this will improve the transaction reading for
> remote
> > > storage.
> > > Overall LGTM, just one minor thought:
> > >
> > > Currently, we only store the `TxnIndexEmpty` bool value in the segment
> > > metadata.
> > > Do you think it is helpful if we store the "least abort start offset in
> > the
> > > segment", and -1 means no txnIndex. So that we can have a way to know
> if
> > we
> > > need to fetch this txn index or not.
> > >
> > > Thanks.
> > > Luke
> > >
> > > On Mon, Sep 9, 2024 at 3:26 PM Kamal Chandraprakash <
> > > kamal.chandraprak...@gmail.com> wrote:
> > >
> > > > Hi all,
> > > >
> > > > If there are no more comments, I'll start a voting thread soon.
> > > >
> > > > Thanks,
> > > > Kamal
> > > >
> > > > On Fri, Sep 6, 2024 at 7:28 PM Kamal Chandraprakash <
> > > > kamal.chandraprak...@gmail.com> wrote:
> > > >
> > > > > Bumping this thread again for review!
> > > > >
> > > > > Reduced the scope of the proposal to minimum. We will be adding
> only
> > > one
> > > > > field (txnIdxEmpty) to the
> > > > > RemoteLogSegmentMetadata event which is backward compatible. PTAL.
> > > > >
> > > > > Thanks,
> > > > > Kamal
> > > > >
> > > > >
> > > > > On Tue, Aug 13, 2024 at 6:33 PM Kamal Chandraprakash <
> > > > > kamal.chandraprak...@gmail.com> wrote:
> > > > >
> > > > >> Bumping this thread for KIP review!
> > > > >>
> > > > >> We can go for the s

[jira] [Created] (KAFKA-17710) Rework UniformHeterogeneousAssignor to improve performance

[Sean Quah (Jira)]

Sean Quah created KAFKA-17710:
-

 Summary: Rework UniformHeterogeneousAssignor to improve performance
 Key: KAFKA-17710
 URL: https://issues.apache.org/jira/browse/KAFKA-17710
 Project: Kafka
  Issue Type: Sub-task
  Components: group-coordinator
Reporter: Sean Quah
Assignee: Sean Quah






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Jenkins build is still unstable: Kafka » Kafka PowerPC Daily » test-powerpc #79

[Apache Jenkins Server]

See 

[jira] [Created] (KAFKA-17708) Fix thread leak: controller-3000-ThrottledChannelReaper-ControllerMutation

[Chia-Ping Tsai (Jira)]

Chia-Ping Tsai created KAFKA-17708:
--

 Summary: Fix thread leak: 
controller-3000-ThrottledChannelReaper-ControllerMutation
 Key: KAFKA-17708
 URL: https://issues.apache.org/jira/browse/KAFKA-17708
 Project: Kafka
  Issue Type: Bug
Reporter: Chia-Ping Tsai
Assignee: Chia-Ping Tsai


{code:java}
org.opentest4j.AssertionFailedError: Thread leak detected: 
controller-3000-ThrottledChannelReaper-ControllerMutation ==> expected:  
but was: 
at 
app//org.apache.kafka.common.test.api.ClusterTestExtensions.afterEach(ClusterTestExtensions.java:159)
at 
java.base@17.0.12/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
at 
java.base@17.0.12/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
at 
java.base@17.0.12/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:179)
at 
java.base@17.0.12/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
at 
java.base@17.0.12/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625)
at 
java.base@17.0.12/java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:762)
at 
java.base@17.0.12/java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:276)
at 
java.base@17.0.12/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625)
at 
java.base@17.0.12/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
at 
java.base@17.0.12/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
at 
java.base@17.0.12/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
at 
java.base@17.0.12/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
at 
java.base@17.0.12/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at 
java.base@17.0.12/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:596)
at java.base@17.0.12/java.util.ArrayList.forEach(ArrayList.java:1511)
at java.base@17.0.12/java.util.ArrayList.forEach(ArrayList.java:1511) 
{code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-17692) Remove KafkaServer references in streams tests

[Chia-Ping Tsai (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-17692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chia-Ping Tsai resolved KAFKA-17692.

Fix Version/s: 4.0.0
   Resolution: Fixed

> Remove KafkaServer references in streams tests
> --
>
> Key: KAFKA-17692
> URL: https://issues.apache.org/jira/browse/KAFKA-17692
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Mickael Maison
>Assignee: Mickael Maison
>Priority: Major
> Fix For: 4.0.0
>
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)