Re: [PR] KAFKA-10789: Streamlining Tests in ChangeLoggingKeyValueBytesStoreTest [kafka]

2025-02-16 Thread via GitHub


leaf-soba commented on PR #18816:
URL: https://github.com/apache/kafka/pull/18816#issuecomment-2661797786

   Hi @chia7712, in the CI process, there are three flaky tests that are 
unrelated to this PR. Could you let me know how to re-run the CI process to get 
them to pass?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



[jira] [Commented] (KAFKA-18332) Fix KafkaRaftClient complexity

2025-02-16 Thread Ryan Ye (Jira)


[ 
https://issues.apache.org/jira/browse/KAFKA-18332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17927585#comment-17927585
 ] 

Ryan Ye commented on KAFKA-18332:
-

Hi can I take this issue?

> Fix KafkaRaftClient complexity
> --
>
> Key: KAFKA-18332
> URL: https://issues.apache.org/jira/browse/KAFKA-18332
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Alyssa Huang
>Assignee: Jose Armando Garcia Sancio
>Priority: Major
>
> Remove suppressions for ClassDataAbstractionCoupling and JavaNCSS
> KafkaRaftClientTest also has suppressions for JavaNCSS (in suppressions.xml), 
> ClassDataAbstractionCoupling, and ClassFanOutComplexity



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[PR] KAFKA-18332: fix ClassDataAbstractionCoupling problem in KafkaRaftClientTest [kafka]

2025-02-16 Thread via GitHub


leaf-soba opened a new pull request, #18926:
URL: https://github.com/apache/kafka/pull/18926

   - extract a unit test named `KafkaRaftClientClusterAuthTest` to reduce the 
number of imported class
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



Re: [PR] KAFKA-18461: Fix potential NPE in setDelta after map is erased [kafka]

2025-02-16 Thread via GitHub


leaf-soba commented on PR #18684:
URL: https://github.com/apache/kafka/pull/18684#issuecomment-2661834192

   Hi @ijuma, could you please take a look?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



[jira] [Created] (KAFKA-18810) Flaky test: ClientIdQuotaTest.testQuotaOverrideDelete

2025-02-16 Thread Andrew Schofield (Jira)
Andrew Schofield created KAFKA-18810:


 Summary: Flaky test: ClientIdQuotaTest.testQuotaOverrideDelete
 Key: KAFKA-18810
 URL: https://issues.apache.org/jira/browse/KAFKA-18810
 Project: Kafka
  Issue Type: Bug
Reporter: Andrew Schofield


Failed in 
https://github.com/apache/kafka/actions/runs/13329397255/job/37230139655?pr=18864
 and more than 20 failures in the past month.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[PR] Mark testQuotaOverrideDelete as flaky [kafka]

2025-02-16 Thread via GitHub


AndrewJSchofield opened a new pull request, #18925:
URL: https://github.com/apache/kafka/pull/18925

   ClientIdQuotaTest.testQuotaOverrideDelete is flaky. Marking this test as 
flaky.
   
   ### Committer Checklist (excluded from commit message)
   - [ ] Verify design and implementation 
   - [ ] Verify test coverage and CI build status
   - [ ] Verify documentation (including upgrade notes)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



Re: [PR] KAFKA-18757: Create full-function SimpleAssignor to match KIP-932 description [kafka]

2025-02-16 Thread via GitHub


apoorvmittal10 commented on code in PR #18864:
URL: https://github.com/apache/kafka/pull/18864#discussion_r1957418602


##
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/assignor/SimpleAssignor.java:
##
@@ -72,36 +77,178 @@ private GroupAssignment assignHomogenous(
 if (subscribeTopicIds.isEmpty())
 return new GroupAssignment(Collections.emptyMap());
 
-Map> targetPartitions = computeTargetPartitions(
+// Subscribed topic partitions for the share group.
+List targetPartitions = computeTargetPartitions(
 subscribeTopicIds, subscribedTopicDescriber);
 
-return new 
GroupAssignment(groupSpec.memberIds().stream().collect(Collectors.toMap(
-Function.identity(), memberId -> new 
MemberAssignmentImpl(targetPartitions;
+// The current assignment from topic partition to members.
+Map> currentAssignment = 
currentAssignment(groupSpec);
+return newAssignmentHomogeneous(groupSpec, subscribeTopicIds, 
targetPartitions, currentAssignment);
 }
 
 private GroupAssignment assignHeterogeneous(
 GroupSpec groupSpec,
 SubscribedTopicDescriber subscribedTopicDescriber
 ) {
-Map members = new HashMap<>();
+Map> memberToPartitionsSubscription = 
new HashMap<>();
 for (String memberId : groupSpec.memberIds()) {
 MemberSubscription spec = groupSpec.memberSubscription(memberId);
 if (spec.subscribedTopicIds().isEmpty())
 continue;
 
-Map> targetPartitions = computeTargetPartitions(
+// Subscribed topic partitions for the share group member.
+List targetPartitions = computeTargetPartitions(
 spec.subscribedTopicIds(), subscribedTopicDescriber);
+memberToPartitionsSubscription.put(memberId, targetPartitions);
+}
+
+// The current assignment from topic partition to members.
+Map> currentAssignment = 
currentAssignment(groupSpec);
+return newAssignmentHeterogeneous(groupSpec, 
memberToPartitionsSubscription, currentAssignment);
+}
 
-members.put(memberId, new MemberAssignmentImpl(targetPartitions));
+// Get the current assignment for subscribed topic partitions to share 
group members.

Review Comment:
   Please write the method comments as javadoc.



##
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/assignor/SimpleAssignor.java:
##
@@ -111,12 +258,26 @@ private Map> computeTargetPartitions(
 );
 }
 
-Set partitions = new HashSet<>();
 for (int i = 0; i < numPartitions; i++) {
-partitions.add(i);
+targetPartitions.add(new TargetPartition(topicId, i));
 }
-targetPartitions.put(topicId, partitions);
 });
 return targetPartitions;
 }
+
+record TargetPartition(Uuid topicId, int partition) {

Review Comment:
   Why we didn't use `org.apache.kafka.server.common.TopicIdPartition`, seems 
same?



##
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/assignor/SimpleAssignor.java:
##
@@ -111,12 +292,41 @@ private Map> computeTargetPartitions(
 );
 }
 
-Set partitions = new HashSet<>();
 for (int i = 0; i < numPartitions; i++) {
-partitions.add(i);
+targetPartitions.add(new TargetPartition(topicId, i));
 }
-targetPartitions.put(topicId, partitions);
 });
 return targetPartitions;
 }
+
+static class TargetPartition {
+Uuid topicId;
+int partition;

Review Comment:
   This can be a record class.



##
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/assignor/SimpleAssignor.java:
##
@@ -72,36 +77,178 @@ private GroupAssignment assignHomogenous(
 if (subscribeTopicIds.isEmpty())
 return new GroupAssignment(Collections.emptyMap());

Review Comment:
   Can we please move to Map.of now?



##
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/assignor/SimpleAssignor.java:
##
@@ -111,12 +258,26 @@ private Map> computeTargetPartitions(
 );
 }
 
-Set partitions = new HashSet<>();
 for (int i = 0; i < numPartitions; i++) {
-partitions.add(i);
+targetPartitions.add(new TargetPartition(topicId, i));
 }
-targetPartitions.put(topicId, partitions);
 });
 return targetPartitions;
 }
+
+record TargetPartition(Uuid topicId, int partition) {
+
+@Override
+public boolean equals(Object o) {
+if (this == o) return true;
+if (o == null || getClass() != o.getClass()) return false;
+TargetPartition that = (TargetPartitio

Re: [PR] KAFKA-18757: Create full-function SimpleAssignor to match KIP-932 description [kafka]

2025-02-16 Thread via GitHub


apoorvmittal10 commented on code in PR #18864:
URL: https://github.com/apache/kafka/pull/18864#discussion_r1954315981


##
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/assignor/SimpleAssignor.java:
##
@@ -111,12 +292,41 @@ private Map> computeTargetPartitions(
 );
 }
 
-Set partitions = new HashSet<>();
 for (int i = 0; i < numPartitions; i++) {
-partitions.add(i);
+targetPartitions.add(new TargetPartition(topicId, i));
 }
-targetPartitions.put(topicId, partitions);
 });
 return targetPartitions;
 }
+
+static class TargetPartition {
+Uuid topicId;
+int partition;

Review Comment:
   This can be a record class.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



[jira] [Assigned] (KAFKA-18673) Provide means to gracefully update Producer transational id mapping state in case of lasting inactivity

2025-02-16 Thread Alex Tran (Jira)


 [ 
https://issues.apache.org/jira/browse/KAFKA-18673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alex Tran reassigned KAFKA-18673:
-

Assignee: Alex Tran

> Provide means to gracefully update Producer transational id mapping state in 
> case of lasting inactivity
> ---
>
> Key: KAFKA-18673
> URL: https://issues.apache.org/jira/browse/KAFKA-18673
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Danil Shkodin
>Assignee: Alex Tran
>Priority: Major
>
> Consider adding some method of preventing a transactional _Producer_ instance 
> from expiring, please.
> Currently, for ones that run services 24/7 that write transactional messages 
> to Kafka very sparsely there are several options to keep the program highly 
> available.
> The first being what Spring  
> [does|https://docs.spring.io/spring-kafka/reference/kafka/transactions.html#overview]:
>  rotating transactional producers at intervals lower than the expiration 
> timeout.
> {code:java}
> void fixTransactionalIdExpiration() {
>   try {
> producer.close(timeout);
>   } catch (Exception error) {
> logger.warn("...", error);
>   }
>   producer = null;
>   try {
> producer = new KafkaProducer<>(settings);
>   } catch (Exception error) {
> logger.warn("...", error);
> // handle failure
> return;
>   }
>   try {
> producer.initTransactions();
>   } catch (Exception error) {
> logger.warn("...", error);
> // close producer and clean up, handle failure
> return;
>   }
> }{code}
> The other similar one is to also act periodically, but to just write an empty 
> record transactionally instead of reconnecting.
> {code:java}
> void fixTransactionalIdExpiration() {
>   try {
> producer.beginTransaction();
> var topic = "project_prefix.__dummy_topic";
> var message = new ProducerRecord<>(topic, (String) null, (String) null);
> producer.send(message);
> producer.abortTransaction();
> // or producer.commitTransaction(); does not matter
>   } catch (Exception error) {
> logger.warn("...", error);
>     // handle failure
>   }
> }{code}
> Personally, I do not like the necessity of introducing a service topic. This 
> inelegance overweights reconnection troubles for me.
> Suprisingly, producing an empty transaction does not prevent expiration. 
> [Probably|https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/producer/internals/TransactionManager.java#L840],
>  there is a guard in the transactional manager that would prevent actual 
> updates to the transactinal producer mappings if there is nothing to write.
> {code:java}
> void fixTransactionalIdExpiration() {
>   // does not work
>   // producer will still get fenced upon transactional.id.expiration.ms
>   try {
> producer.beginTransaction();
> producer.commitTransaction();
>   } catch (Exception error) {}
> } {code}
> Worth noting that client code may execute one of these periodic fixes 
> conditionally, only if there was no activity, meaning there were no 
> successful _send()_ or _sendOffsetsToTransaction()_ for, say, 6 - 24 hours.
> The last and obvious one is to let it fail and react to the error.
> {code:java}
> void sendMessage(Message message) {
>   try {
> producer.beginTransaction();
> producer.send(message.to());
> producer.commitTransaction();
>   } catch (InvalidPidMappingException error) {
> // reconnect, retry
>   } catch (Exception error) {
> // handle failure
>   }
> }{code}
> Having a dedicated method that explicitly reflects the intent to refresh 
> producer _transactional.id_ would line up with the _Consumer_ polling 
> mechanism and manifest to new kafka-clients users that lasting transactional 
> _Producer_ inactivity should be addressed.
> This issue search optimization:
> InvalidPidMappingException: The producer attempted to use a producer id which 
> is not currently assigned to its transactional id
> transactional.id.expiration.ms
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Assigned] (KAFKA-7952) Consider to switch to in-memory stores in test whenever possible

2025-02-16 Thread Alex Tran (Jira)


 [ 
https://issues.apache.org/jira/browse/KAFKA-7952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alex Tran reassigned KAFKA-7952:


Assignee: (was: Alex Tran)

> Consider to switch to in-memory stores in test whenever possible
> 
>
> Key: KAFKA-7952
> URL: https://issues.apache.org/jira/browse/KAFKA-7952
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams, unit tests
>Reporter: Matthias J. Sax
>Priority: Major
>  Labels: beginner, newbie
>
> We observed that tests can be very slow using default RocksDB stores (cf. 
> KAFKA-7933).
> We should consider to switch to in-memory stores whenever possible to reduce 
> test runtime.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


Re: [PR] KAFKA-18784: Fix ConsumerWithLegacyMessageFormatIntegrationTest [kafka]

2025-02-16 Thread via GitHub


chia7712 commented on code in PR #18889:
URL: https://github.com/apache/kafka/pull/18889#discussion_r1957459755


##
core/src/test/scala/integration/kafka/api/ConsumerWithLegacyMessageFormatIntegrationTest.scala:
##
@@ -18,16 +18,43 @@ package kafka.api
 
 import kafka.utils.TestInfoUtils
 import org.apache.kafka.common.TopicPartition
+import org.apache.kafka.common.compress.Compression
+import org.apache.kafka.common.record.{AbstractRecords, CompressionType, 
MemoryRecords, RecordBatch, RecordVersion, SimpleRecord, TimestampType}
 import org.junit.jupiter.api.Assertions.{assertEquals, assertNull, 
assertThrows}
 import org.junit.jupiter.params.ParameterizedTest
 import org.junit.jupiter.params.provider.MethodSource
 
+import java.nio.ByteBuffer
 import java.util
 import java.util.{Collections, Optional}
 import scala.jdk.CollectionConverters._
 
 class ConsumerWithLegacyMessageFormatIntegrationTest extends 
AbstractConsumerTest {
 
+  private def appendLegacyRecords(numRecords: Int, tp: TopicPartition, 
startingTimestamp: Long, brokerId: Int): Unit = {
+val records = (0 until numRecords).map { i =>
+  new SimpleRecord(startingTimestamp + i, s"key $i".getBytes, s"value 
$i".getBytes)
+}
+val buffer = 
ByteBuffer.allocate(AbstractRecords.estimateSizeInBytes(RecordBatch.MAGIC_VALUE_V1,
 CompressionType.NONE, records.asJava))
+val builder = MemoryRecords.builder(buffer, RecordBatch.MAGIC_VALUE_V1, 
Compression.of(CompressionType.NONE).build,

Review Comment:
   Could you please add test for v0 also? in v0 we don't write the timestamp, 
so searching record by timestamp should return null.



##
core/src/test/scala/integration/kafka/api/ConsumerWithLegacyMessageFormatIntegrationTest.scala:
##
@@ -80,12 +114,12 @@ class ConsumerWithLegacyMessageFormatIntegrationTest 
extends AbstractConsumerTes
 val timestampTopic2P0 = timestampOffsets.get(new TopicPartition(topic2, 0))
 assertEquals(40, timestampTopic2P0.offset)
 assertEquals(40, timestampTopic2P0.timestamp)
-assertEquals(Optional.of(0), timestampTopic2P0.leaderEpoch)
+assertEquals(Optional.empty, timestampTopic2P0.leaderEpoch)

Review Comment:
   yes, we don't set the leader epoch for the old message format.



##
core/src/test/scala/integration/kafka/api/ConsumerWithLegacyMessageFormatIntegrationTest.scala:
##
@@ -51,8 +78,15 @@ class ConsumerWithLegacyMessageFormatIntegrationTest extends 
AbstractConsumerTes
 for (topic <- List(topic1, topic2, topic3)) {
   for (part <- 0 until numParts) {
 val tp = new TopicPartition(topic, part)
-// In sendRecords(), each message will have key, value and timestamp 
equal to the sequence number.
-sendRecords(producer, numRecords = 100, tp, startingTimestamp = 0)
+// In sendRecords() and sendLegacyRecords(), each message will have 
key, value and timestamp equal to the sequence number.
+if (topic == topic2) {
+  // append legacy records to topic2
+  appendLegacyRecords(100, tp, 0, part)
+} else {
+  println("sendRecords")

Review Comment:
   please remove this line



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



Re: [PR] Update streams compat table 3.8 [kafka]

2025-02-16 Thread via GitHub


github-actions[bot] commented on PR #17850:
URL: https://github.com/apache/kafka/pull/17850#issuecomment-2661892400

   This PR is being marked as stale since it has not had any activity in 90 
days. If you
   would like to keep this PR alive, please leave a comment asking for a 
review. If the PR has 
   merge conflicts, update it with the latest from the base branch.
   
   If you are having difficulty finding a reviewer, please reach out on the 
   [mailing list](https://kafka.apache.org/contact).
   
   If this PR is no longer valid or desired, please feel free to close it. If 
no activity
   occurs in the next 30 days, it will be automatically closed.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



Re: [PR] KAFKA-18767: Add client side config check for shareConsumer [kafka]

2025-02-16 Thread via GitHub


TaiJuWu commented on code in PR #18850:
URL: https://github.com/apache/kafka/pull/18850#discussion_r1957292877


##
clients/src/main/java/org/apache/kafka/clients/consumer/GroupProtocol.java:
##
@@ -23,7 +23,10 @@ public enum GroupProtocol {
 CLASSIC("CLASSIC"),
 
 /** Consumer group protocol */
-CONSUMER("CONSUMER");
+CONSUMER("CONSUMER"),
+
+/** Share group protocol */
+SHARE("SHARE");

Review Comment:
   c4ed6834d1b9f05c5e38247c5e0904d84150508e does not work since 
`GroupRebalanceConfig` has default value.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



Re: [PR] KAFKA-18767: Add client side config check for shareConsumer [kafka]

2025-02-16 Thread via GitHub


TaiJuWu commented on code in PR #18850:
URL: https://github.com/apache/kafka/pull/18850#discussion_r1957292877


##
clients/src/main/java/org/apache/kafka/clients/consumer/GroupProtocol.java:
##
@@ -23,7 +23,10 @@ public enum GroupProtocol {
 CLASSIC("CLASSIC"),
 
 /** Consumer group protocol */
-CONSUMER("CONSUMER");
+CONSUMER("CONSUMER"),
+
+/** Share group protocol */
+SHARE("SHARE");

Review Comment:
   cf239aefa82053cb7a5f89e5e12335829a2b4e76 does not work since 
`GroupRebalanceConfig` has default value so we need to do all check within 
`ConsumerConfig`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



[jira] [Commented] (KAFKA-15332) Eligible Leader Replicas

2025-02-16 Thread Stanislav Kozlovski (Jira)


[ 
https://issues.apache.org/jira/browse/KAFKA-15332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17927535#comment-17927535
 ] 

Stanislav Kozlovski commented on KAFKA-15332:
-

[~calvinliu] [~dajac] [~jhooper] can you confirm whether 4.0 is just shipping 
the ELR part of the KIP?

My understanding is the behavior without unclear recovery is basically a slight 
improvement from today's state, where we have and can use the ELR set with the 
highest watermark thing, but in case the set is empty - we're back to today's 
behavior more or less

> Eligible Leader Replicas
> 
>
> Key: KAFKA-15332
> URL: https://issues.apache.org/jira/browse/KAFKA-15332
> Project: Kafka
>  Issue Type: New Feature
>Reporter: Calvin Liu
>Assignee: Calvin Liu
>Priority: Major
>
> A root ticket for the KIP-966
> The KIP has accepted 
> [https://cwiki.apache.org/confluence/display/KAFKA/KIP-966%3A+Eligible+Leader+Replicas]
> The delivery is divided into 2 parts, ELR and unclean recovery.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Commented] (KAFKA-18735) the zk-related path `` is still existent in quota system, such as `ConfigEntity#name` and metrics tags

2025-02-16 Thread Chia-Ping Tsai (Jira)


[ 
https://issues.apache.org/jira/browse/KAFKA-18735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17927504#comment-17927504
 ] 

Chia-Ping Tsai commented on KAFKA-18735:


{quote}
If there is some API that is setting ConfigEntity#name to  for default 
quotas, I think that's just straightforwardly a bug that should be fixed in 4.0.
{quote}

there is no such case. Using "name=" can't create default quota. We 
can add a test to ensure that scenario.

Currently, the issue I have observed is `ConfigEntity` used by callback. It 
return " as name when it is the default quota. In the APIs doc, it 
says empty string will be returned if it is the default quota. However, both 
 and empty string are weird to me as the entity returned by 
Admin#describeClientQuotas is using `null` to represent default quota.

We can keep returning  to the callback to keep compatibility. Or we 
can add a "small" breaking change to the callback to align the value of entity 
to "null" 


> the zk-related path `` is still existent in quota system, such as 
> `ConfigEntity#name` and metrics tags
> ---
>
> Key: KAFKA-18735
> URL: https://issues.apache.org/jira/browse/KAFKA-18735
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Chia-Ping Tsai
>Assignee: Ming-Yen Chung
>Priority: Minor
> Fix For: 4.1.0, 4.0.1
>
>
> the `` is a specific "flag" used by zk path to represent the 
> "default" quota. We don't expose it on Admin as we use `null` to represent 
> the default name [0], and in KAFKA-16411 we assumes `` should be 
> removed from 4.0 with the end of ZK mode [1]
> However, the `` is still used in two "public APIs"
> 1. ConfigEntity#name return  for default quotas. The docs say it 
> return empty string, but that is incorrect. see [2]
> -"client-id" -> "" and "user" -> "" are used in metrics tag 
> [3]-
>  
> [0] 
> https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/common/quota/ClientQuotaEntity.java#L44
> [1] 
> https://github.com/apache/kafka/blob/trunk/server-common/src/main/java/org/apache/kafka/server/config/ZooKeeperInternals.java#L24
> [2] 
> https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/server/ClientQuotaManager.scala#L79
> [3] 
> https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/server/ClientQuotaManager.scala#L479



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Commented] (KAFKA-18808) add test to ensure the name= is not equal to default quota

2025-02-16 Thread Chia-Ping Tsai (Jira)


[ 
https://issues.apache.org/jira/browse/KAFKA-18808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17927505#comment-17927505
 ] 

Chia-Ping Tsai commented on KAFKA-18808:



{code:java}
Map entries = new HashMap<>();
entries.put("client-id", "");
try (var adminClient = 
Admin.create(Collections.singletonMap("bootstrap.servers", 
cluster.bootstrapServers( {
adminClient.alterClientQuotas(Collections.singletonList(new 
ClientQuotaAlteration(
new 
org.apache.kafka.common.quota.ClientQuotaEntity(entries),
Collections.singleton(new 
ClientQuotaAlteration.Op("producer_byte_rate", 1D).all().get();
TimeUnit.SECONDS.sleep(2);
Map> entityMapMap = adminClient.describeClientQuotas(

ClientQuotaFilter.containsOnly(Collections.singletonList(ClientQuotaFilterComponent.ofDefaultEntity("client-id"
.entities().get();
Assertions.assertEquals(0, entityMapMap.size());
}
{code}


> add test to ensure the name= is not equal to default quota
> ---
>
> Key: KAFKA-18808
> URL: https://issues.apache.org/jira/browse/KAFKA-18808
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Chia-Ping Tsai
>Assignee: Chia-Ping Tsai
>Priority: Major
> Fix For: 4.1.0
>
>
> see discussion in KAFKA-18735 - the test should include following check.
> 1. Using name= does not create default quota
> 2. the returned entity should have name=
> 2. the filter `ClientQuotaFilterComponent.ofDefaultEntity` should return 
> nothing



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Updated] (KAFKA-18735) the zk-related path `` is still used by quota callback

2025-02-16 Thread Chia-Ping Tsai (Jira)


 [ 
https://issues.apache.org/jira/browse/KAFKA-18735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chia-Ping Tsai updated KAFKA-18735:
---
Summary: the zk-related path `` is still used by quota callback  
(was: the zk-related path `` is still existent in quota system, such 
as `ConfigEntity#name` and metrics tags)

> the zk-related path `` is still used by quota callback
> ---
>
> Key: KAFKA-18735
> URL: https://issues.apache.org/jira/browse/KAFKA-18735
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Chia-Ping Tsai
>Assignee: Ming-Yen Chung
>Priority: Minor
> Fix For: 4.1.0, 4.0.1
>
>
> the `` is a specific "flag" used by zk path to represent the 
> "default" quota. We don't expose it on Admin as we use `null` to represent 
> the default name [0], and in KAFKA-16411 we assumes `` should be 
> removed from 4.0 with the end of ZK mode [1]
> However, the `` is still used in two "public APIs"
> 1. ConfigEntity#name return  for default quotas. The docs say it 
> return empty string, but that is incorrect. see [2]
> -"client-id" -> "" and "user" -> "" are used in metrics tag 
> [3]-
>  
> [0] 
> https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/common/quota/ClientQuotaEntity.java#L44
> [1] 
> https://github.com/apache/kafka/blob/trunk/server-common/src/main/java/org/apache/kafka/server/config/ZooKeeperInternals.java#L24
> [2] 
> https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/server/ClientQuotaManager.scala#L79
> [3] 
> https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/server/ClientQuotaManager.scala#L479



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


Re: [PR] KAFKA-14484: Decouple UnifiedLog and RemoteLogManager [kafka]

2025-02-16 Thread via GitHub


kamalcph commented on code in PR #18460:
URL: https://github.com/apache/kafka/pull/18460#discussion_r1957373391


##
storage/src/main/java/org/apache/kafka/storage/internals/log/AsyncOffsetReader.java:
##
@@ -0,0 +1,47 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.storage.internals.log;
+
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.record.FileRecords;
+import org.apache.kafka.storage.internals.epoch.LeaderEpochFileCache;
+
+import java.util.Optional;
+import java.util.function.Supplier;
+
+/**
+ * Interface used to decouple UnifiedLog and RemoteLogManager.
+ */
+public interface AsyncOffsetReader {
+
+/**
+ * Retrieve the offset for the specified timestamp. UnifiedLog may call 
this method when handling OffsetRequest > 0

Review Comment:
   > when handling OffsetRequest > 0
   
   I couldn't understand, What does it mean? Is it referring to timestamp > 0? 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



Re: [PR] KAFKA-14484: Decouple UnifiedLog and RemoteLogManager [kafka]

2025-02-16 Thread via GitHub


kamalcph commented on code in PR #18460:
URL: https://github.com/apache/kafka/pull/18460#discussion_r1957373391


##
storage/src/main/java/org/apache/kafka/storage/internals/log/AsyncOffsetReader.java:
##
@@ -0,0 +1,47 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.storage.internals.log;
+
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.record.FileRecords;
+import org.apache.kafka.storage.internals.epoch.LeaderEpochFileCache;
+
+import java.util.Optional;
+import java.util.function.Supplier;
+
+/**
+ * Interface used to decouple UnifiedLog and RemoteLogManager.
+ */
+public interface AsyncOffsetReader {
+
+/**
+ * Retrieve the offset for the specified timestamp. UnifiedLog may call 
this method when handling OffsetRequest > 0

Review Comment:
   > when handling OffsetRequest > 0
   
   I couldn't understand, What does it mean?



##
core/src/main/scala/kafka/log/UnifiedLog.scala:
##
@@ -1235,7 +1234,7 @@ class UnifiedLog(@volatile var logStartOffset: Long,
*   All special timestamp offset results are returned 
immediately irrespective of the remote storage.
* 
*/
-  def fetchOffsetByTimestamp(targetTimestamp: Long, remoteLogManager: 
Option[RemoteLogManager] = None): OffsetResultHolder = {
+  def fetchOffsetByTimestamp(targetTimestamp: Long, remoteLogManager: 
Optional[AsyncOffsetReader] = Optional.empty): OffsetResultHolder = {

Review Comment:
   nit: 
   
   Can we rename the variable `remoteLogManager` to `remoteOffsetReader`? As, 
the other methods in RemoteLogManager are no longer accessible. 



##
core/src/main/java/kafka/log/remote/RemoteLogManager.java:
##
@@ -664,10 +664,10 @@ private Optional maybeLeaderEpoch(int 
leaderEpoch) {
 
 public AsyncOffsetReadFutureHolder 
asyncOffsetRead(

Review Comment:
   Add `@Override` to the method.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



Re: [PR] KAFKA-18641: AsyncKafkaConsumer could lose records with auto offset commit [kafka]

2025-02-16 Thread via GitHub


junrao commented on code in PR #18737:
URL: https://github.com/apache/kafka/pull/18737#discussion_r1957375625


##
clients/src/main/java/org/apache/kafka/clients/consumer/internals/events/ApplicationEventProcessor.java:
##
@@ -206,13 +206,25 @@ public void process(ApplicationEvent event) {
 }
 
 private void process(final PollEvent event) {
+// Trigger a reconciliation that can safely commit offsets if needed 
to rebalance,
+// as we're processing before any new fetching starts in the app thread
+
requestManagers.consumerMembershipManager.ifPresent(consumerMembershipManager ->
+consumerMembershipManager.maybeReconcile(true));
 if (requestManagers.commitRequestManager.isPresent()) {
-requestManagers.commitRequestManager.ifPresent(m -> 
m.updateAutoCommitTimer(event.pollTimeMs()));
+CommitRequestManager commitRequestManager = 
requestManagers.commitRequestManager.get();
+commitRequestManager.updateTimerAndMaybeCommit(event.pollTimeMs());
+// all commit request generation points have been passed,
+// so it's safe to notify the app thread could proceed and start 
fetching
+event.future().complete(null);

Review Comment:
   Technically, the event hasn't been fully processed yet. So, it's a bit weird 
to complete the event midway. Perhaps we could introduce a separate future like 
`reconcileAndAutoCommit` and wait for that.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



Re: [PR] KAFKA-18806: Resource leak in MemoryRecordsBuilder in RecordsUtil [kafka]

2025-02-16 Thread via GitHub


xelathan commented on PR #18919:
URL: https://github.com/apache/kafka/pull/18919#issuecomment-2661578406

   Build finished with unrelated test failure in `ConsumerCoordinatorTest > 
testOutdatedCoordinatorAssignment `
   Issue is tracked with 
[KAFKA-15900](https://issues.apache.org/jira/browse/KAFKA-15900)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



Re: [PR] KAFKA-18757: Create full-function SimpleAssignor to match KIP-932 description [kafka]

2025-02-16 Thread via GitHub


AndrewJSchofield commented on code in PR #18864:
URL: https://github.com/apache/kafka/pull/18864#discussion_r1957425369


##
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/assignor/SimpleAssignor.java:
##
@@ -72,36 +77,178 @@ private GroupAssignment assignHomogenous(
 if (subscribeTopicIds.isEmpty())
 return new GroupAssignment(Collections.emptyMap());
 
-Map> targetPartitions = computeTargetPartitions(
+// Subscribed topic partitions for the share group.
+List targetPartitions = computeTargetPartitions(
 subscribeTopicIds, subscribedTopicDescriber);
 
-return new 
GroupAssignment(groupSpec.memberIds().stream().collect(Collectors.toMap(
-Function.identity(), memberId -> new 
MemberAssignmentImpl(targetPartitions;
+// The current assignment from topic partition to members.
+Map> currentAssignment = 
currentAssignment(groupSpec);
+return newAssignmentHomogeneous(groupSpec, subscribeTopicIds, 
targetPartitions, currentAssignment);
 }
 
 private GroupAssignment assignHeterogeneous(
 GroupSpec groupSpec,
 SubscribedTopicDescriber subscribedTopicDescriber
 ) {
-Map members = new HashMap<>();
+Map> memberToPartitionsSubscription = 
new HashMap<>();
 for (String memberId : groupSpec.memberIds()) {
 MemberSubscription spec = groupSpec.memberSubscription(memberId);
 if (spec.subscribedTopicIds().isEmpty())
 continue;
 
-Map> targetPartitions = computeTargetPartitions(
+// Subscribed topic partitions for the share group member.
+List targetPartitions = computeTargetPartitions(
 spec.subscribedTopicIds(), subscribedTopicDescriber);
+memberToPartitionsSubscription.put(memberId, targetPartitions);
+}
+
+// The current assignment from topic partition to members.
+Map> currentAssignment = 
currentAssignment(groupSpec);
+return newAssignmentHeterogeneous(groupSpec, 
memberToPartitionsSubscription, currentAssignment);
+}
 
-members.put(memberId, new MemberAssignmentImpl(targetPartitions));
+// Get the current assignment for subscribed topic partitions to share 
group members.
+private Map> currentAssignment(GroupSpec 
groupSpec) {
+Map> assignment = new HashMap<>();
+
+for (String member : groupSpec.memberIds()) {
+Map> assignedTopicPartitions = 
groupSpec.memberAssignment(member).partitions();
+assignedTopicPartitions.forEach((topicId, partitions) -> 
partitions.forEach(
+partition -> assignment.computeIfAbsent(new 
TargetPartition(topicId, partition), k -> new ArrayList<>()).add(member)));
 }
+return assignment;
+}
+
+private GroupAssignment newAssignmentHomogeneous(
+GroupSpec groupSpec,
+Set subscribeTopicIds,
+List targetPartitions,
+Map> currentAssignment) {
+
+Map> newAssignment = new HashMap<>();
+// Step 1: Hash member IDs to partitions.
+memberHashAssignment(targetPartitions, groupSpec.memberIds(), 
newAssignment);
+
+// Step 2: Round-robin assignment for unassigned partitions which do 
not have members already assigned in the current assignment.
+Set assignedPartitions = new 
HashSet<>(newAssignment.keySet());
+List unassignedPartitions = targetPartitions.stream()
+.filter(targetPartition -> 
!assignedPartitions.contains(targetPartition))
+.filter(targetPartition -> 
!currentAssignment.containsKey(targetPartition))
+.collect(Collectors.toList());
+
+roundRobinAssignment(groupSpec.memberIds(), unassignedPartitions, 
newAssignment);
+
+Map> finalAssignment = new HashMap<>();
+
+// When combining current assignment, we need to only consider the 
topics in current assignment that are also being
+// subscribed in the new assignment as well.
+currentAssignment.forEach((targetPartition, members) -> {
+if (subscribeTopicIds.contains(targetPartition.topicId))
+members.forEach(member -> {
+if (groupSpec.memberIds().contains(member))
+finalAssignment.computeIfAbsent(member, k -> new 
HashSet<>()).add(targetPartition);
+});
+});
+newAssignment.forEach((targetPartition, members) -> 
members.forEach(member ->
+finalAssignment.computeIfAbsent(member, k -> new 
HashSet<>()).add(targetPartition)));
+
+return groupAssignment(finalAssignment, groupSpec.memberIds());
+}
+
+private GroupAssignment newAssignmentHeterogeneous(
+GroupSpec groupSpec,
+Map> memberToPartitionsSubscription,
+Map> currentAssignment) {
+
+// Exhaustive set of all subscribed topic partitions.
+Set targetPartition

Re: [PR] KAFKA-18757: Create full-function SimpleAssignor to match KIP-932 description [kafka]

2025-02-16 Thread via GitHub


apoorvmittal10 commented on code in PR #18864:
URL: https://github.com/apache/kafka/pull/18864#discussion_r1957427989


##
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/assignor/SimpleAssignor.java:
##
@@ -72,36 +77,178 @@ private GroupAssignment assignHomogenous(
 if (subscribeTopicIds.isEmpty())
 return new GroupAssignment(Collections.emptyMap());
 
-Map> targetPartitions = computeTargetPartitions(
+// Subscribed topic partitions for the share group.
+List targetPartitions = computeTargetPartitions(
 subscribeTopicIds, subscribedTopicDescriber);
 
-return new 
GroupAssignment(groupSpec.memberIds().stream().collect(Collectors.toMap(
-Function.identity(), memberId -> new 
MemberAssignmentImpl(targetPartitions;
+// The current assignment from topic partition to members.
+Map> currentAssignment = 
currentAssignment(groupSpec);
+return newAssignmentHomogeneous(groupSpec, subscribeTopicIds, 
targetPartitions, currentAssignment);
 }
 
 private GroupAssignment assignHeterogeneous(
 GroupSpec groupSpec,
 SubscribedTopicDescriber subscribedTopicDescriber
 ) {
-Map members = new HashMap<>();
+Map> memberToPartitionsSubscription = 
new HashMap<>();
 for (String memberId : groupSpec.memberIds()) {
 MemberSubscription spec = groupSpec.memberSubscription(memberId);
 if (spec.subscribedTopicIds().isEmpty())
 continue;
 
-Map> targetPartitions = computeTargetPartitions(
+// Subscribed topic partitions for the share group member.
+List targetPartitions = computeTargetPartitions(
 spec.subscribedTopicIds(), subscribedTopicDescriber);
+memberToPartitionsSubscription.put(memberId, targetPartitions);
+}
+
+// The current assignment from topic partition to members.
+Map> currentAssignment = 
currentAssignment(groupSpec);
+return newAssignmentHeterogeneous(groupSpec, 
memberToPartitionsSubscription, currentAssignment);
+}
 
-members.put(memberId, new MemberAssignmentImpl(targetPartitions));
+// Get the current assignment for subscribed topic partitions to share 
group members.
+private Map> currentAssignment(GroupSpec 
groupSpec) {
+Map> assignment = new HashMap<>();
+
+for (String member : groupSpec.memberIds()) {
+Map> assignedTopicPartitions = 
groupSpec.memberAssignment(member).partitions();
+assignedTopicPartitions.forEach((topicId, partitions) -> 
partitions.forEach(
+partition -> assignment.computeIfAbsent(new 
TargetPartition(topicId, partition), k -> new ArrayList<>()).add(member)));
 }
+return assignment;
+}
+
+private GroupAssignment newAssignmentHomogeneous(
+GroupSpec groupSpec,
+Set subscribeTopicIds,
+List targetPartitions,
+Map> currentAssignment) {
+
+Map> newAssignment = new HashMap<>();
+// Step 1: Hash member IDs to partitions.
+memberHashAssignment(targetPartitions, groupSpec.memberIds(), 
newAssignment);
+
+// Step 2: Round-robin assignment for unassigned partitions which do 
not have members already assigned in the current assignment.
+Set assignedPartitions = new 
HashSet<>(newAssignment.keySet());
+List unassignedPartitions = targetPartitions.stream()
+.filter(targetPartition -> 
!assignedPartitions.contains(targetPartition))
+.filter(targetPartition -> 
!currentAssignment.containsKey(targetPartition))
+.collect(Collectors.toList());
+
+roundRobinAssignment(groupSpec.memberIds(), unassignedPartitions, 
newAssignment);
+
+Map> finalAssignment = new HashMap<>();
+
+// When combining current assignment, we need to only consider the 
topics in current assignment that are also being
+// subscribed in the new assignment as well.
+currentAssignment.forEach((targetPartition, members) -> {
+if (subscribeTopicIds.contains(targetPartition.topicId))
+members.forEach(member -> {
+if (groupSpec.memberIds().contains(member))
+finalAssignment.computeIfAbsent(member, k -> new 
HashSet<>()).add(targetPartition);
+});
+});
+newAssignment.forEach((targetPartition, members) -> 
members.forEach(member ->
+finalAssignment.computeIfAbsent(member, k -> new 
HashSet<>()).add(targetPartition)));
+
+return groupAssignment(finalAssignment, groupSpec.memberIds());
+}
+
+private GroupAssignment newAssignmentHeterogeneous(
+GroupSpec groupSpec,
+Map> memberToPartitionsSubscription,
+Map> currentAssignment) {
+
+// Exhaustive set of all subscribed topic partitions.
+Set targetPartitions 

Re: [PR] KAFKA-18757: Create full-function SimpleAssignor to match KIP-932 description [kafka]

2025-02-16 Thread via GitHub


AndrewJSchofield commented on PR #18864:
URL: https://github.com/apache/kafka/pull/18864#issuecomment-2661633762

   Marked failed test as flaky in https://github.com/apache/kafka/pull/18925.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



Re: [PR] KAFKA-18804: Remove slf4j warning when using tool script [kafka]

2025-02-16 Thread via GitHub


frankvicky commented on PR #18918:
URL: https://github.com/apache/kafka/pull/18918#issuecomment-2661387467

   Hi @m1a2st 
   You should see the warning of slf4j, but it is different from this one.
   ```
   > Task :core:genProtocolErrorDocs
   SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
   SLF4J: Defaulting to no-operation (NOP) logger implementation
   SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further 
details.
   ```
   The above warning is handled by 
[KAFKA-18752](https://issues.apache.org/jira/browse/KAFKA-18752)
   
   > And why don't we exclude the Log4j dependency in the core module? It is 
defined in `build.gradle` at line 1137.
   
   We could do that, but it is not necessary. The warning states that slf4j 
detects multiple bindings in projects, so removing one is enough. In this 
situation, it is easiest to exclude one in the tools module.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



Re: [PR] KAFKA-18767: Add client side config check for shareConsumer [kafka]

2025-02-16 Thread via GitHub


TaiJuWu commented on code in PR #18850:
URL: https://github.com/apache/kafka/pull/18850#discussion_r1957292877


##
clients/src/main/java/org/apache/kafka/clients/consumer/GroupProtocol.java:
##
@@ -23,7 +23,10 @@ public enum GroupProtocol {
 CLASSIC("CLASSIC"),
 
 /** Consumer group protocol */
-CONSUMER("CONSUMER");
+CONSUMER("CONSUMER"),
+
+/** Share group protocol */
+SHARE("SHARE");

Review Comment:
   cf239aefa82053cb7a5f89e5e12335829a2b4e76 does not work since 
`GroupRebalanceConfig` has default value.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



Re: [PR] KAFKA-18804: Remove slf4j warning when using tool script [kafka]

2025-02-16 Thread via GitHub


m1a2st commented on PR #18918:
URL: https://github.com/apache/kafka/pull/18918#issuecomment-2661393738

   > The above warning is handled by 
[KAFKA-18752](https://issues.apache.org/jira/browse/KAFKA-18752)
   
   I'm not referring to this warning. I tested the steps below, and no warning 
messages appeared.
   ```
   ./gradlew clean
   ./gradlew releaseTarGz
   tar -zxvf $(find ./core/build/distributions/ -maxdepth 1 -type f \( -iname 
"kafka*tgz" ! -iname "*sit*" \)) -C /tmp/opt/kafka --strip-components=1
   ./bin/kafka-storage.sh -h
   usage: kafka-storage [-h] 
{info,format,version-mapping,feature-dependencies,random-uuid} ...
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



Re: [PR] KAFKA-18804: Remove slf4j warning when using tool script [kafka]

2025-02-16 Thread via GitHub


frankvicky commented on PR #18918:
URL: https://github.com/apache/kafka/pull/18918#issuecomment-2661398712

   At the source code root, you need to complete the `build` or `releaseTarGz` 
at least first. This will generate a `dependant-libs-2.13.15` directory. 
   Currently, only `core` and `tools` use `releaseOnly`. The 
`copyDependantLibs` tasks for these two modules will copy the `releaseOnly` 
dependencies to `dependant-libs-2.13.15`.
   Therefore, when running tool scripts under the source code root, it will 
find two `slf4-impl` jars.
   So this is indeed a dev-specific issue.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



Re: [PR] KAFKA-18767: Add client side config check for shareConsumer [kafka]

2025-02-16 Thread via GitHub


TaiJuWu commented on PR #18850:
URL: https://github.com/apache/kafka/pull/18850#issuecomment-2661404879

   > Thanks for the PR. Just one overall comment about why 
`GroupProtocol.SHARE` was defined. The `GROUP_PROTOCOL_CONFIG` is used to 
choose between which group protocol is used for the regular `KafkaConsumer`. 
It's basically the consumer switch for KIP-848. I think that's different than 
`GroupProtocol.SHARE` which I don't think should exist.
   
   I think the current state is the most simple solution, WDYT?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



Re: [PR] KAFKA-18767: Add client side config check for shareConsumer [kafka]

2025-02-16 Thread via GitHub


TaiJuWu commented on code in PR #18850:
URL: https://github.com/apache/kafka/pull/18850#discussion_r1957242661


##
clients/src/main/java/org/apache/kafka/clients/consumer/GroupProtocol.java:
##
@@ -23,7 +23,10 @@ public enum GroupProtocol {
 CLASSIC("CLASSIC"),
 
 /** Consumer group protocol */
-CONSUMER("CONSUMER");
+CONSUMER("CONSUMER"),
+
+/** Share group protocol */
+SHARE("SHARE");

Review Comment:
   Hi @AndrewJSchofield , I add config checker to `GroupRebalanceConfig`.
   Is this change ok?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



Re: [PR] KAFKA-18767: Add client side config check for shareConsumer [kafka]

2025-02-16 Thread via GitHub


TaiJuWu commented on code in PR #18850:
URL: https://github.com/apache/kafka/pull/18850#discussion_r1957292877


##
clients/src/main/java/org/apache/kafka/clients/consumer/GroupProtocol.java:
##
@@ -23,7 +23,10 @@ public enum GroupProtocol {
 CLASSIC("CLASSIC"),
 
 /** Consumer group protocol */
-CONSUMER("CONSUMER");
+CONSUMER("CONSUMER"),
+
+/** Share group protocol */
+SHARE("SHARE");

Review Comment:
   This method does not work since `GroupRebalanceConfig` has default value.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



[jira] [Commented] (KAFKA-18805) Consumer Heartbeat closed change should be locked

2025-02-16 Thread Jira


[ 
https://issues.apache.org/jira/browse/KAFKA-18805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17927473#comment-17927473
 ] 

黃竣陽 commented on KAFKA-18805:
-

Hello [~showuon], if you won't work on this, may I take this?

> Consumer Heartbeat closed change should be locked
> -
>
> Key: KAFKA-18805
> URL: https://issues.apache.org/jira/browse/KAFKA-18805
> Project: Kafka
>  Issue Type: Bug
>Reporter: Luke Chen
>Priority: Major
>  Labels: newbie
>
> This `this.closed = true` is not wrapped inside the lock, which could cause 
> the change is not visible in another thread.
>  
> https://github.com/apache/kafka/blob/e330f0bf2570a27811fa20a2f446b101a7a656f3/clients/src/main/java/org/apache/kafka/clients/consumer/internals/AbstractCoordinator.java#L1580



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Commented] (KAFKA-18806) Resource leak in MemoryRecordsBuilder in RecordsUtil

2025-02-16 Thread TengYao Chi (Jira)


[ 
https://issues.apache.org/jira/browse/KAFKA-18806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17927474#comment-17927474
 ] 

TengYao Chi commented on KAFKA-18806:
-

Hi [~showuon] 

May I take over this one ?

> Resource leak in MemoryRecordsBuilder in RecordsUtil
> 
>
> Key: KAFKA-18806
> URL: https://issues.apache.org/jira/browse/KAFKA-18806
> Project: Kafka
>  Issue Type: Bug
>Reporter: Luke Chen
>Priority: Major
>  Labels: newbie
>
> We forgot to close the MemoryRecordsBuilder resource (which contains streams) 
> here: 
> https://github.com/apache/kafka/blob/e330f0bf2570a27811fa20a2f446b101a7a656f3/clients/src/main/java/org/apache/kafka/common/record/RecordsUtil.java#L92.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Created] (KAFKA-18806) Resource leak in MemoryRecordsBuilder in RecordsUtil

2025-02-16 Thread Luke Chen (Jira)
Luke Chen created KAFKA-18806:
-

 Summary: Resource leak in MemoryRecordsBuilder in RecordsUtil
 Key: KAFKA-18806
 URL: https://issues.apache.org/jira/browse/KAFKA-18806
 Project: Kafka
  Issue Type: Bug
Reporter: Luke Chen


We forgot to close the MemoryRecordsBuilder resource (which contains streams) 
here: 
https://github.com/apache/kafka/blob/e330f0bf2570a27811fa20a2f446b101a7a656f3/clients/src/main/java/org/apache/kafka/common/record/RecordsUtil.java#L92.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Assigned] (KAFKA-18806) Resource leak in MemoryRecordsBuilder in RecordsUtil

2025-02-16 Thread Alex Tran (Jira)


 [ 
https://issues.apache.org/jira/browse/KAFKA-18806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alex Tran reassigned KAFKA-18806:
-

Assignee: Alex Tran

> Resource leak in MemoryRecordsBuilder in RecordsUtil
> 
>
> Key: KAFKA-18806
> URL: https://issues.apache.org/jira/browse/KAFKA-18806
> Project: Kafka
>  Issue Type: Bug
>Reporter: Luke Chen
>Assignee: Alex Tran
>Priority: Major
>  Labels: newbie
>
> We forgot to close the MemoryRecordsBuilder resource (which contains streams) 
> here: 
> https://github.com/apache/kafka/blob/e330f0bf2570a27811fa20a2f446b101a7a656f3/clients/src/main/java/org/apache/kafka/common/record/RecordsUtil.java#L92.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


Re: [PR] KAFKA-18804: Remove slf4j warning when using tool script [kafka]

2025-02-16 Thread via GitHub


Yunyung commented on PR #18918:
URL: https://github.com/apache/kafka/pull/18918#issuecomment-2661333030

   This should be backported to 4.0, as the same issue happens in 4.0.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



[jira] [Commented] (KAFKA-18805) Consumer Heartbeat closed change should be locked

2025-02-16 Thread Luke Chen (Jira)


[ 
https://issues.apache.org/jira/browse/KAFKA-18805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17927477#comment-17927477
 ] 

Luke Chen commented on KAFKA-18805:
---

[~m1a2st] , assigned to you. Thanks.

> Consumer Heartbeat closed change should be locked
> -
>
> Key: KAFKA-18805
> URL: https://issues.apache.org/jira/browse/KAFKA-18805
> Project: Kafka
>  Issue Type: Bug
>Reporter: Luke Chen
>Assignee: 黃竣陽
>Priority: Major
>  Labels: newbie
>
> This `this.closed = true` is not wrapped inside the lock, which could cause 
> the change is not visible in another thread.
>  
> https://github.com/apache/kafka/blob/e330f0bf2570a27811fa20a2f446b101a7a656f3/clients/src/main/java/org/apache/kafka/clients/consumer/internals/AbstractCoordinator.java#L1580



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Commented] (KAFKA-18806) Resource leak in MemoryRecordsBuilder in RecordsUtil

2025-02-16 Thread Luke Chen (Jira)


[ 
https://issues.apache.org/jira/browse/KAFKA-18806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17927478#comment-17927478
 ] 

Luke Chen commented on KAFKA-18806:
---

[~frankvicky] , it looks like [~alectric] is picking is up. [~alectric] , are 
you going to work on it?

> Resource leak in MemoryRecordsBuilder in RecordsUtil
> 
>
> Key: KAFKA-18806
> URL: https://issues.apache.org/jira/browse/KAFKA-18806
> Project: Kafka
>  Issue Type: Bug
>Reporter: Luke Chen
>Assignee: Alex Tran
>Priority: Major
>  Labels: newbie
>
> We forgot to close the MemoryRecordsBuilder resource (which contains streams) 
> here: 
> https://github.com/apache/kafka/blob/e330f0bf2570a27811fa20a2f446b101a7a656f3/clients/src/main/java/org/apache/kafka/common/record/RecordsUtil.java#L92.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Assigned] (KAFKA-18805) Consumer Heartbeat closed change should be locked

2025-02-16 Thread Luke Chen (Jira)


 [ 
https://issues.apache.org/jira/browse/KAFKA-18805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Chen reassigned KAFKA-18805:
-

Assignee: 黃竣陽

> Consumer Heartbeat closed change should be locked
> -
>
> Key: KAFKA-18805
> URL: https://issues.apache.org/jira/browse/KAFKA-18805
> Project: Kafka
>  Issue Type: Bug
>Reporter: Luke Chen
>Assignee: 黃竣陽
>Priority: Major
>  Labels: newbie
>
> This `this.closed = true` is not wrapped inside the lock, which could cause 
> the change is not visible in another thread.
>  
> https://github.com/apache/kafka/blob/e330f0bf2570a27811fa20a2f446b101a7a656f3/clients/src/main/java/org/apache/kafka/clients/consumer/internals/AbstractCoordinator.java#L1580



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Created] (KAFKA-18807) Group coordinator under-reports thread idle time

2025-02-16 Thread Sean Quah (Jira)
Sean Quah created KAFKA-18807:
-

 Summary: Group coordinator under-reports thread idle time
 Key: KAFKA-18807
 URL: https://issues.apache.org/jira/browse/KAFKA-18807
 Project: Kafka
  Issue Type: Bug
  Components: group-coordinator
Reporter: Sean Quah
Assignee: Sean Quah


When events arrive at the event processor with less than 
1/group.coordinator.threads ms between them, the thread idle time is rounded 
down to 0 because of integer division.

eg. when the group coordinator has 8 threads, and it receives >1k offset commit 
requests per second evenly split across shards, the thread idle time metric 
will be under-reported.

https://github.com/apache/kafka/blob/2353a7c5081b53402d5092afd255e2bdf7aa2d20/coordinator-common/src/main/java/org/apache/kafka/coordinator/common/runtime/MultiThreadedEventProcessor.java#L141



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[PR] KAFKA-18806: Resource leak in MemoryRecordsBuilder in RecordsUtil [kafka]

2025-02-16 Thread via GitHub


xelathan opened a new pull request, #18919:
URL: https://github.com/apache/kafka/pull/18919

   MemoryRecordsBuilder not closed in RecordsUtil.
   Closed implicitly with `try`
   
   https://issues.apache.org/jira/browse/KAFKA-18806
   
   ### Committer Checklist (excluded from commit message)
   - [ ] Verify design and implementation 
   - [ ] Verify test coverage and CI build status
   - [ ] Verify documentation (including upgrade notes)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



[jira] [Commented] (KAFKA-18806) Resource leak in MemoryRecordsBuilder in RecordsUtil

2025-02-16 Thread Alex Tran (Jira)


[ 
https://issues.apache.org/jira/browse/KAFKA-18806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17927481#comment-17927481
 ] 

Alex Tran commented on KAFKA-18806:
---

[~showuon] , yes I am working on it.

> Resource leak in MemoryRecordsBuilder in RecordsUtil
> 
>
> Key: KAFKA-18806
> URL: https://issues.apache.org/jira/browse/KAFKA-18806
> Project: Kafka
>  Issue Type: Bug
>Reporter: Luke Chen
>Assignee: Alex Tran
>Priority: Major
>  Labels: newbie
>
> We forgot to close the MemoryRecordsBuilder resource (which contains streams) 
> here: 
> https://github.com/apache/kafka/blob/e330f0bf2570a27811fa20a2f446b101a7a656f3/clients/src/main/java/org/apache/kafka/common/record/RecordsUtil.java#L92.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Commented] (KAFKA-18806) Resource leak in MemoryRecordsBuilder in RecordsUtil

2025-02-16 Thread TengYao Chi (Jira)


[ 
https://issues.apache.org/jira/browse/KAFKA-18806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17927484#comment-17927484
 ] 

TengYao Chi commented on KAFKA-18806:
-

No worried

I will help review this one

> Resource leak in MemoryRecordsBuilder in RecordsUtil
> 
>
> Key: KAFKA-18806
> URL: https://issues.apache.org/jira/browse/KAFKA-18806
> Project: Kafka
>  Issue Type: Bug
>Reporter: Luke Chen
>Assignee: Alex Tran
>Priority: Major
>  Labels: newbie
>
> We forgot to close the MemoryRecordsBuilder resource (which contains streams) 
> here: 
> https://github.com/apache/kafka/blob/e330f0bf2570a27811fa20a2f446b101a7a656f3/clients/src/main/java/org/apache/kafka/common/record/RecordsUtil.java#L92.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[PR] KAFKA-18805: Consumer Heartbeat closed change should be locked [kafka]

2025-02-16 Thread via GitHub


m1a2st opened a new pull request, #18920:
URL: https://github.com/apache/kafka/pull/18920

   Jira: https://issues.apache.org/jira/browse/KAFKA-18805
   
   Using synchronized block to avoid the locked
   
   ### Committer Checklist (excluded from commit message)
   - [ ] Verify design and implementation 
   - [ ] Verify test coverage and CI build status
   - [ ] Verify documentation (including upgrade notes)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



[jira] [Resolved] (KAFKA-18805) Consumer Heartbeat closed change should be locked

2025-02-16 Thread Luke Chen (Jira)


 [ 
https://issues.apache.org/jira/browse/KAFKA-18805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Chen resolved KAFKA-18805.
---
Fix Version/s: 4.1.0
   Resolution: Fixed

> Consumer Heartbeat closed change should be locked
> -
>
> Key: KAFKA-18805
> URL: https://issues.apache.org/jira/browse/KAFKA-18805
> Project: Kafka
>  Issue Type: Bug
>Reporter: Luke Chen
>Assignee: 黃竣陽
>Priority: Major
>  Labels: newbie
> Fix For: 4.1.0
>
>
> This `this.closed = true` is not wrapped inside the lock, which could cause 
> the change is not visible in another thread.
>  
> https://github.com/apache/kafka/blob/e330f0bf2570a27811fa20a2f446b101a7a656f3/clients/src/main/java/org/apache/kafka/clients/consumer/internals/AbstractCoordinator.java#L1580



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


Re: [PR] KAFKA-16918: TestUtils#assertFutureThrows should use future.get with timeout [kafka]

2025-02-16 Thread via GitHub


chia7712 commented on code in PR #18891:
URL: https://github.com/apache/kafka/pull/18891#discussion_r1957734129


##
clients/src/test/java/org/apache/kafka/test/TestUtils.java:
##
@@ -580,21 +583,28 @@ public static Set 
generateRandomTopicPartitions(int numTopic, in
 }
 
 /**
- * Assert that a future raises an expected exception cause type. Return 
the exception cause
- * if the assertion succeeds; otherwise raise AssertionError.
+ * Assert that a future raises an expected exception cause type.
+ * This method will wait for the future to complete or timeout(15000 
milliseconds) after a default period.
  *
  * @param future The future to await
  * @param exceptionCauseClass Class of the expected exception cause
  * @param  Exception cause type parameter
  * @return The caught exception cause
  */
 public static  T assertFutureThrows(Future future, 
Class exceptionCauseClass) {
-ExecutionException exception = assertThrows(ExecutionException.class, 
future::get);
-Throwable cause = exception.getCause();
-assertEquals(exceptionCauseClass, cause.getClass(),
-"Expected a " + exceptionCauseClass.getSimpleName() + " exception, 
but got " +
-cause.getClass().getSimpleName());
-return exceptionCauseClass.cast(exception.getCause());
+try {
+future.get(DEFAULT_MAX_WAIT_MS, TimeUnit.MILLISECONDS);
+fail("Should throw expected exception " + 
exceptionCauseClass.getSimpleName() + " but nothing was thrown.");
+} catch (InterruptedException | ExecutionException | 
CancellationException e) {
+Throwable cause = e instanceof ExecutionException ? e.getCause() : 
e;
+return assertInstanceOf(
+exceptionCauseClass, cause,
+"Expected " + exceptionCauseClass.getSimpleName() + ", but got 
" + cause
+);
+} catch (TimeoutException e) {
+fail("Future is not completed within " + DEFAULT_MAX_WAIT_MS + " 
millisecond.");
+}
+throw new RuntimeException("Future should throw expected exception but 
unexpected error happened.");

Review Comment:
   Could you please include the "unexpected error"?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



[jira] [Assigned] (KAFKA-18810) Flaky test: ClientIdQuotaTest.testQuotaOverrideDelete

2025-02-16 Thread Chia-Ping Tsai (Jira)


 [ 
https://issues.apache.org/jira/browse/KAFKA-18810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chia-Ping Tsai reassigned KAFKA-18810:
--

Assignee: Chia-Ping Tsai

> Flaky test: ClientIdQuotaTest.testQuotaOverrideDelete
> -
>
> Key: KAFKA-18810
> URL: https://issues.apache.org/jira/browse/KAFKA-18810
> Project: Kafka
>  Issue Type: Bug
>Reporter: Andrew Schofield
>Assignee: Chia-Ping Tsai
>Priority: Major
>
> Failed in 
> https://github.com/apache/kafka/actions/runs/13329397255/job/37230139655?pr=18864
>  and more than 20 failures in the past month.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


Re: [PR] Mark testQuotaOverrideDelete as flaky [kafka]

2025-02-16 Thread via GitHub


chia7712 merged PR #18925:
URL: https://github.com/apache/kafka/pull/18925


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



Re: [PR] KAFKA-18784: Fix ConsumerWithLegacyMessageFormatIntegrationTest [kafka]

2025-02-16 Thread via GitHub


chia7712 commented on code in PR #18889:
URL: https://github.com/apache/kafka/pull/18889#discussion_r1957730744


##
core/src/test/scala/integration/kafka/api/ConsumerWithLegacyMessageFormatIntegrationTest.scala:
##
@@ -18,16 +18,43 @@ package kafka.api
 
 import kafka.utils.TestInfoUtils
 import org.apache.kafka.common.TopicPartition
+import org.apache.kafka.common.compress.Compression
+import org.apache.kafka.common.record.{AbstractRecords, CompressionType, 
MemoryRecords, RecordBatch, RecordVersion, SimpleRecord, TimestampType}
 import org.junit.jupiter.api.Assertions.{assertEquals, assertNull, 
assertThrows}
 import org.junit.jupiter.params.ParameterizedTest
 import org.junit.jupiter.params.provider.MethodSource
 
+import java.nio.ByteBuffer
 import java.util
 import java.util.{Collections, Optional}
 import scala.jdk.CollectionConverters._
 
 class ConsumerWithLegacyMessageFormatIntegrationTest extends 
AbstractConsumerTest {
 
+  private def appendLegacyRecords(numRecords: Int, tp: TopicPartition, 
startingTimestamp: Long, brokerId: Int): Unit = {
+val records = (0 until numRecords).map { i =>
+  new SimpleRecord(startingTimestamp + i, s"key $i".getBytes, s"value 
$i".getBytes)
+}
+val buffer = 
ByteBuffer.allocate(AbstractRecords.estimateSizeInBytes(RecordBatch.MAGIC_VALUE_V0,
 CompressionType.NONE, records.asJava))
+val builder = MemoryRecords.builder(buffer, RecordBatch.MAGIC_VALUE_V0, 
Compression.of(CompressionType.NONE).build,

Review Comment:
   Could you please keep tests for both v0 and v1?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



Re: [PR] KAFKA-18712: Move Endpoint to server module [kafka]

2025-02-16 Thread via GitHub


chia7712 commented on PR #18803:
URL: https://github.com/apache/kafka/pull/18803#issuecomment-2662303442

   @TaiJuWu could you please rebase code?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



Re: [PR] KAFKA-18805: Consumer Heartbeat closed change should be locked [kafka]

2025-02-16 Thread via GitHub


showuon merged PR #18920:
URL: https://github.com/apache/kafka/pull/18920


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



Re: [PR] KAFKA-18213: Convert CustomQuotaCallbackTest to KRaft [kafka]

2025-02-16 Thread via GitHub


chia7712 closed pull request #18126: KAFKA-18213: Convert 
CustomQuotaCallbackTest to KRaft
URL: https://github.com/apache/kafka/pull/18126


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



Re: [PR] KAFKA-18213: Convert CustomQuotaCallbackTest to KRaft [kafka]

2025-02-16 Thread via GitHub


chia7712 commented on PR #18126:
URL: https://github.com/apache/kafka/pull/18126#issuecomment-2662245203

   close as duplicate to #18196


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



[jira] [Created] (KAFKA-18808) add test to ensure the name= is not equal to default quota

2025-02-16 Thread Chia-Ping Tsai (Jira)
Chia-Ping Tsai created KAFKA-18808:
--

 Summary: add test to ensure the name= is not equal to 
default quota
 Key: KAFKA-18808
 URL: https://issues.apache.org/jira/browse/KAFKA-18808
 Project: Kafka
  Issue Type: Sub-task
Reporter: Chia-Ping Tsai
Assignee: Chia-Ping Tsai
 Fix For: 4.1.0


see discussion in KAFKA-18735 - the test should include following check.

1. Using name= does not create default quota
2. the returned entity should have name=
2. the filter `ClientQuotaFilterComponent.ofDefaultEntity` should return nothing



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


Re: [PR] KAFKA-18803: The acls would appear at the wrong level of the metadata shell "tree" [kafka]

2025-02-16 Thread via GitHub


Yunyung commented on PR #18916:
URL: https://github.com/apache/kafka/pull/18916#issuecomment-2661447763

   > 1. Manually test this patch by creating some ACLs and showing the output 
from metadata shell
   
   Before:
   ```
   $ ./bin/kafka-metadata-shell.sh --snapshot 
/tmp/kraft-combined-logs/__cluster_metadata-0/.log
   Loading...
   [2025-02-16 13:58:14,107] ERROR Ignoring control record with type 
KRAFT_VERSION at offset 1 (org.apache.kafka.metadata.util.SnapshotFileReader)
   [2025-02-16 13:58:14,110] ERROR Ignoring control record with type 
KRAFT_VOTERS at offset 2 (org.apache.kafka.metadata.util.SnapshotFileReader)
   Starting...
   [ Kafka Metadata Shell ]
   >> ls image/acls/
   5Mu_djJ5QZaw79Fc3URLfw
   >>
   ```
   
   After:
   ```
   $ ./bin/kafka-metadata-shell.sh --snapshot 
/tmp/kraft-combined-logs/__cluster_metadata-0/.log
   Loading...
   [2025-02-16 21:59:06,250] ERROR Ignoring control record with type 
KRAFT_VERSION at offset 1 (org.apache.kafka.metadata.util.SnapshotFileReader)
   [2025-02-16 21:59:06,251] ERROR Ignoring control record with type 
KRAFT_VOTERS at offset 2 (org.apache.kafka.metadata.util.SnapshotFileReader)
   Starting...
   [ Kafka Metadata Shell ]
   >> ls image/acls/
   byId
   >> ls image/acls/byId/
   cnTIY8C5R5-PIm0EKywItg
   >> 
   
   ```
   
   > 2. Add a unit test for the MetadataImageNode to ensure we see the "acls" 
node correctly
   
   Done.
   
   Thanks for the review, PTAL.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



[jira] [Commented] (KAFKA-18808) add test to ensure the name= is not equal to default quota

2025-02-16 Thread Nick Guo (Jira)


[ 
https://issues.apache.org/jira/browse/KAFKA-18808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17927508#comment-17927508
 ] 

Nick Guo commented on KAFKA-18808:
--

Hi [~chia7712] ,may I take this issue? Thanks! :)

> add test to ensure the name= is not equal to default quota
> ---
>
> Key: KAFKA-18808
> URL: https://issues.apache.org/jira/browse/KAFKA-18808
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Chia-Ping Tsai
>Assignee: Chia-Ping Tsai
>Priority: Major
> Fix For: 4.1.0
>
>
> see discussion in KAFKA-18735 - the test should include following check.
> 1. Using name= does not create default quota
> 2. the returned entity should have name=
> 2. the filter `ClientQuotaFilterComponent.ofDefaultEntity` should return 
> nothing



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[PR] KAFKA-18809: Set min in sync replicas for __share_group_state. [kafka]

2025-02-16 Thread via GitHub


smjn opened a new pull request, #18922:
URL: https://github.com/apache/kafka/pull/18922

   * The `share.coordinator.state.topic.min.isr` config defined in 
`ShareCoordinatorConfig` was not being used in the `AutoTopicCreationManager`. 
   * The `AutoTopicCreationManager` calls the 
`ShareCoordinatorService.shareGroupStateTopicConfigs` to configs for the topic 
to create.
   * The method `ShareCoordinatorService.shareGroupStateTopicConfigs` was not 
setting the supplied config value for `share.coordinator.state.topic.min.isr` 
to `min.insync.replicas`.
   * In this PR, we remedy the situation by setting the value.
   * A test has been added to `ShareCoordinatorServiceTest` so that this is not 
repeated for any configs.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



[jira] [Commented] (KAFKA-18809) share.coordinator.state.topic.min.isr is not applied

2025-02-16 Thread Sushant Mahajan (Jira)


[ 
https://issues.apache.org/jira/browse/KAFKA-18809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17927537#comment-17927537
 ] 

Sushant Mahajan commented on KAFKA-18809:
-

[~chia7712] 

Raised a fix https://github.com/apache/kafka/pull/18922

> share.coordinator.state.topic.min.isr is not applied 
> -
>
> Key: KAFKA-18809
> URL: https://issues.apache.org/jira/browse/KAFKA-18809
> Project: Kafka
>  Issue Type: Bug
>Reporter: Chia-Ping Tsai
>Assignee: Chia-Ping Tsai
>Priority: Blocker
> Fix For: 4.1.0
>
>
> https://github.com/apache/kafka/blob/e330f0bf2570a27811fa20a2f446b101a7a656f3/share-coordinator/src/main/java/org/apache/kafka/coordinator/share/ShareCoordinatorService.java#L233
> in the `shareGroupStateTopicConfigs` we ignore the 
> `share.coordinator.state.topic.min.isr`



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[PR] MINOR: deflake EligibleLeaderReplicasIntegrationTest [kafka]

2025-02-16 Thread via GitHub


CalvinConfluent opened a new pull request, #18923:
URL: https://github.com/apache/kafka/pull/18923

   (no comment)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



Re: [PR] KAFKA-18801 Remove ClusterGenerator and revise docs of ClusterTemplate [kafka]

2025-02-16 Thread via GitHub


JimmyWang6 commented on code in PR #18907:
URL: https://github.com/apache/kafka/pull/18907#discussion_r1957392043


##
test-common/test-common-internal-api/src/main/java/org/apache/kafka/common/test/api/ClusterTemplate.java:
##
@@ -30,9 +30,9 @@
 
 /**
  * Used to indicate that a test should call the method given by {@link 
#value()} to generate a number of
- * cluster configurations. The method specified by the value should accept a 
single argument of the type
- * {@link ClusterGenerator}. Any return value from the method is ignored. A 
test invocation
- * will be generated for each {@link ClusterConfig} provided to the 
ClusterGenerator instance.
+ * cluster configurations. The method specified by the value does not accept 
any arguments.

Review Comment:
   Hi @chia7712, If the method is not defined as static, an exception is thrown 
as following:
   `Cannot invoke non-static method [private 
java.util.List 
org.apache.kafka.tools.consumer.group.DeleteConsumerGroupsTest.generator()] on 
a null target.
   org.junit.platform.commons.PreconditionViolationException: Cannot invoke 
non-static method [private 
java.util.List 
org.apache.kafka.tools.consumer.group.DeleteConsumerGroupsTest.generator()] on 
a null target.
at 
org.apache.kafka.common.test.junit.ClusterTestExtensions.generateClusterConfiguration(ClusterTestExtensions.java:235)
   `
   I haven't thought of a good way to prevent the method that returns 
List from being non-static yet. Any suggestions? I have added 
the sample code.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



[jira] [Commented] (KAFKA-15900) Flaky test: testOutdatedCoordinatorAssignment() – org.apache.kafka.clients.consumer.internals.EagerConsumerCoordinatorTest

2025-02-16 Thread Alex Tran (Jira)


[ 
https://issues.apache.org/jira/browse/KAFKA-15900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17927555#comment-17927555
 ] 

Alex Tran commented on KAFKA-15900:
---

I am also seeing it fail in unrelated 
[PR|https://github.com/apache/kafka/pull/18919] regarding cleaning up resources 
in `MemoryRecordsBuilder`. 

[https://github.com/apache/kafka/actions/runs/13353687223/job/37293878236?pr=18919]

> Flaky test: testOutdatedCoordinatorAssignment() – 
> org.apache.kafka.clients.consumer.internals.EagerConsumerCoordinatorTest
> --
>
> Key: KAFKA-15900
> URL: https://issues.apache.org/jira/browse/KAFKA-15900
> Project: Kafka
>  Issue Type: Bug
>Reporter: Apoorv Mittal
>Assignee: TaiJuWu
>Priority: Major
>
> Build run: 
> [https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka-pr/detail/PR-14620/15/tests/]
>  
> {code:java}
> org.opentest4j.AssertionFailedError: expected: <1> but was: 
> <2>Stacktraceorg.opentest4j.AssertionFailedError: expected: <1> but was: <2>  
> at 
> app//org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
>at 
> app//org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
>at 
> app//org.junit.jupiter.api.AssertEquals.failNotEqual(AssertEquals.java:197)  
> at 
> app//org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:150)  
> at 
> app//org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:145)  
> at app//org.junit.jupiter.api.Assertions.assertEquals(Assertions.java:527)
>   at 
> app//org.apache.kafka.clients.consumer.internals.ConsumerCoordinatorTest.testOutdatedCoordinatorAssignment(ConsumerCoordinatorTest.java:1067)
> at 
> java.base@21.0.1/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
>  at java.base@21.0.1/java.lang.reflect.Method.invoke(Method.java:580)at 
> app//org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:728)
>   at 
> app//org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
>at 
> app//org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
>  at 
> app//org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:156)
> at 
> app//org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:147)
>   at 
> app//org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:86)
>at 
> app//org.junit.jupiter.engine.execution.InterceptingExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(InterceptingExecutableInvoker.java:103)
> at 
> app//org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.lambda$invoke$0(InterceptingExecutableInvoker.java:93)
>  at 
> app//org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
> at 
> app//org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
>at 
> app//org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45)
> at 
> app//org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37)
>  {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Created] (KAFKA-18809) share.coordinator.state.topic.min.isr is not applied

2025-02-16 Thread Chia-Ping Tsai (Jira)
Chia-Ping Tsai created KAFKA-18809:
--

 Summary: share.coordinator.state.topic.min.isr is not applied 
 Key: KAFKA-18809
 URL: https://issues.apache.org/jira/browse/KAFKA-18809
 Project: Kafka
  Issue Type: Bug
Reporter: Chia-Ping Tsai
Assignee: Chia-Ping Tsai
 Fix For: 4.1.0


https://github.com/apache/kafka/blob/e330f0bf2570a27811fa20a2f446b101a7a656f3/share-coordinator/src/main/java/org/apache/kafka/coordinator/share/ShareCoordinatorService.java#L233

in the `shareGroupStateTopicConfigs` we ignore the 
`share.coordinator.state.topic.min.isr`



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Commented] (KAFKA-18809) share.coordinator.state.topic.min.isr is not applied

2025-02-16 Thread Chia-Ping Tsai (Jira)


[ 
https://issues.apache.org/jira/browse/KAFKA-18809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17927518#comment-17927518
 ] 

Chia-Ping Tsai commented on KAFKA-18809:


[~apoorvmittal10] [~schofielaj] [~sushmahajn] not sure whether you have other 
plan to apply the config. If it is just a bug, I will file a patch for it.

> share.coordinator.state.topic.min.isr is not applied 
> -
>
> Key: KAFKA-18809
> URL: https://issues.apache.org/jira/browse/KAFKA-18809
> Project: Kafka
>  Issue Type: Bug
>Reporter: Chia-Ping Tsai
>Assignee: Chia-Ping Tsai
>Priority: Blocker
> Fix For: 4.1.0
>
>
> https://github.com/apache/kafka/blob/e330f0bf2570a27811fa20a2f446b101a7a656f3/share-coordinator/src/main/java/org/apache/kafka/coordinator/share/ShareCoordinatorService.java#L233
> in the `shareGroupStateTopicConfigs` we ignore the 
> `share.coordinator.state.topic.min.isr`



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Assigned] (KAFKA-18808) add test to ensure the name= is not equal to default quota

2025-02-16 Thread Chia-Ping Tsai (Jira)


 [ 
https://issues.apache.org/jira/browse/KAFKA-18808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chia-Ping Tsai reassigned KAFKA-18808:
--

Assignee: Nick Guo  (was: Chia-Ping Tsai)

> add test to ensure the name= is not equal to default quota
> ---
>
> Key: KAFKA-18808
> URL: https://issues.apache.org/jira/browse/KAFKA-18808
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Chia-Ping Tsai
>Assignee: Nick Guo
>Priority: Major
> Fix For: 4.1.0
>
>
> see discussion in KAFKA-18735 - the test should include following check.
> 1. Using name= does not create default quota
> 2. the returned entity should have name=
> 2. the filter `ClientQuotaFilterComponent.ofDefaultEntity` should return 
> nothing



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


Re: [PR] KAFKA-18801 Remove ClusterGenerator and revise docs of ClusterTemplate [kafka]

2025-02-16 Thread via GitHub


JimmyWang6 commented on code in PR #18907:
URL: https://github.com/apache/kafka/pull/18907#discussion_r1957377060


##
test-common/test-common-internal-api/src/main/java/org/apache/kafka/common/test/api/ClusterTemplate.java:
##
@@ -30,9 +30,9 @@
 
 /**
  * Used to indicate that a test should call the method given by {@link 
#value()} to generate a number of
- * cluster configurations. The method specified by the value should accept a 
single argument of the type
- * {@link ClusterGenerator}. Any return value from the method is ignored. A 
test invocation
- * will be generated for each {@link ClusterConfig} provided to the 
ClusterGenerator instance.
+ * cluster configurations. The method specified by the value does not accept 
any arguments.

Review Comment:
   Thanks for @mumrah and @chia7712 's comments, I will do that according to 
your suggestions.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



[jira] [Commented] (KAFKA-15332) Eligible Leader Replicas

2025-02-16 Thread Calvin Liu (Jira)


[ 
https://issues.apache.org/jira/browse/KAFKA-15332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17927543#comment-17927543
 ] 

Calvin Liu commented on KAFKA-15332:


[~stanislavkozlovski] 4.0 only carries part 1 which enhances data safety when 
no unclean leader election is needed. Typical use cases are network 
partitioning, performance-degraded leaders and clean shutdown replica.

For part 2, it is planned to be implemented this year. 

> Eligible Leader Replicas
> 
>
> Key: KAFKA-15332
> URL: https://issues.apache.org/jira/browse/KAFKA-15332
> Project: Kafka
>  Issue Type: New Feature
>Reporter: Calvin Liu
>Assignee: Calvin Liu
>Priority: Major
>
> A root ticket for the KIP-966
> The KIP has accepted 
> [https://cwiki.apache.org/confluence/display/KAFKA/KIP-966%3A+Eligible+Leader+Replicas]
> The delivery is divided into 2 parts, ELR and unclean recovery.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[PR] [WIP] KAFKA-18562: standardize election/fetch timeout between Unattached and Followers [kafka]

2025-02-16 Thread via GitHub


frankvicky opened a new pull request, #18921:
URL: https://github.com/apache/kafka/pull/18921

   JIRA: KAFKA-18562
   
   ### Committer Checklist (excluded from commit message)
   - [ ] Verify design and implementation 
   - [ ] Verify test coverage and CI build status
   - [ ] Verify documentation (including upgrade notes)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



[jira] [Updated] (KAFKA-18803) The acls would appear at the wrong level of the metadata shell "tree"

2025-02-16 Thread Chia-Ping Tsai (Jira)


 [ 
https://issues.apache.org/jira/browse/KAFKA-18803?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chia-Ping Tsai updated KAFKA-18803:
---
Fix Version/s: 3.9.1
   3.8.2

> The acls would appear at the wrong level of the metadata shell "tree"
> -
>
> Key: KAFKA-18803
> URL: https://issues.apache.org/jira/browse/KAFKA-18803
> Project: Kafka
>  Issue Type: Bug
>Reporter: Chia-Ping Tsai
>Assignee: Jhen-Yung Hsu
>Priority: Blocker
> Fix For: 4.0.0, 3.9.1, 3.8.2
>
>
> see https://github.com/apache/kafka/pull/18845#discussion_r1956525411



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


Re: [PR] KAFKA-18803: The acls would appear at the wrong level of the metadata shell "tree" [kafka]

2025-02-16 Thread via GitHub


chia7712 merged PR #18916:
URL: https://github.com/apache/kafka/pull/18916


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



[jira] [Resolved] (KAFKA-18803) The acls would appear at the wrong level of the metadata shell "tree"

2025-02-16 Thread Chia-Ping Tsai (Jira)


 [ 
https://issues.apache.org/jira/browse/KAFKA-18803?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chia-Ping Tsai resolved KAFKA-18803.

Resolution: Fixed

trunk: 
https://github.com/apache/kafka/commit/d0e516a87207b04f6266c59712775b5f01c1ed87

4.0: 
https://github.com/apache/kafka/commit/8797c903eef894e63fa15710cf446715eceb9198

3.9: 
https://github.com/apache/kafka/commit/0c288599c5f6c3d71e2cb6db76af64f0a88d5b9f

3.8: 
https://github.com/apache/kafka/commit/7d65d0f3fefc9dab7918d77f950408621fc5a667

> The acls would appear at the wrong level of the metadata shell "tree"
> -
>
> Key: KAFKA-18803
> URL: https://issues.apache.org/jira/browse/KAFKA-18803
> Project: Kafka
>  Issue Type: Bug
>Reporter: Chia-Ping Tsai
>Assignee: Jhen-Yung Hsu
>Priority: Blocker
> Fix For: 4.0.0, 3.9.1, 3.8.2
>
>
> see https://github.com/apache/kafka/pull/18845#discussion_r1956525411



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Resolved] (KAFKA-18809) share.coordinator.state.topic.min.isr is not applied

2025-02-16 Thread Chia-Ping Tsai (Jira)


 [ 
https://issues.apache.org/jira/browse/KAFKA-18809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chia-Ping Tsai resolved KAFKA-18809.

Resolution: Fixed

I keep this fix in 4.1, as shared consumer is not in production to 4.0

> share.coordinator.state.topic.min.isr is not applied 
> -
>
> Key: KAFKA-18809
> URL: https://issues.apache.org/jira/browse/KAFKA-18809
> Project: Kafka
>  Issue Type: Bug
>Reporter: Chia-Ping Tsai
>Assignee: Sushant Mahajan
>Priority: Blocker
> Fix For: 4.1.0
>
>
> https://github.com/apache/kafka/blob/e330f0bf2570a27811fa20a2f446b101a7a656f3/share-coordinator/src/main/java/org/apache/kafka/coordinator/share/ShareCoordinatorService.java#L233
> in the `shareGroupStateTopicConfigs` we ignore the 
> `share.coordinator.state.topic.min.isr`



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Updated] (KAFKA-18809) share.coordinator.state.topic.min.isr is not applied

2025-02-16 Thread Chia-Ping Tsai (Jira)


 [ 
https://issues.apache.org/jira/browse/KAFKA-18809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chia-Ping Tsai updated KAFKA-18809:
---
Fix Version/s: 4.1.0
   (was: 4.0.0)

> share.coordinator.state.topic.min.isr is not applied 
> -
>
> Key: KAFKA-18809
> URL: https://issues.apache.org/jira/browse/KAFKA-18809
> Project: Kafka
>  Issue Type: Bug
>Reporter: Chia-Ping Tsai
>Assignee: Sushant Mahajan
>Priority: Blocker
> Fix For: 4.1.0
>
>
> https://github.com/apache/kafka/blob/e330f0bf2570a27811fa20a2f446b101a7a656f3/share-coordinator/src/main/java/org/apache/kafka/coordinator/share/ShareCoordinatorService.java#L233
> in the `shareGroupStateTopicConfigs` we ignore the 
> `share.coordinator.state.topic.min.isr`



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Resolved] (KAFKA-17682) Refactor RaftClusterInvocationContext to eliminate test-common-api dependencies

2025-02-16 Thread Chia-Ping Tsai (Jira)


 [ 
https://issues.apache.org/jira/browse/KAFKA-17682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chia-Ping Tsai resolved KAFKA-17682.

Resolution: Duplicate

resolved by 
https://github.com/apache/kafka/commit/8c0a0e07ced062419cf51c31307e5d168ad9efbc

> Refactor RaftClusterInvocationContext to eliminate test-common-api 
> dependencies
> ---
>
> Key: KAFKA-17682
> URL: https://issues.apache.org/jira/browse/KAFKA-17682
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Chia-Ping Tsai
>Assignee: bboyleonp
>Priority: Major
>
> see https://github.com/apache/kafka/pull/17318/files#r1784648528



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


Re: [PR] KAFKA-18809: Set min in sync replicas for __share_group_state. [kafka]

2025-02-16 Thread via GitHub


chia7712 merged PR #18922:
URL: https://github.com/apache/kafka/pull/18922


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



[jira] [Updated] (KAFKA-18809) share.coordinator.state.topic.min.isr is not applied

2025-02-16 Thread Chia-Ping Tsai (Jira)


 [ 
https://issues.apache.org/jira/browse/KAFKA-18809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chia-Ping Tsai updated KAFKA-18809:
---
Fix Version/s: 4.0.0
   (was: 4.1.0)

> share.coordinator.state.topic.min.isr is not applied 
> -
>
> Key: KAFKA-18809
> URL: https://issues.apache.org/jira/browse/KAFKA-18809
> Project: Kafka
>  Issue Type: Bug
>Reporter: Chia-Ping Tsai
>Assignee: Sushant Mahajan
>Priority: Blocker
> Fix For: 4.0.0
>
>
> https://github.com/apache/kafka/blob/e330f0bf2570a27811fa20a2f446b101a7a656f3/share-coordinator/src/main/java/org/apache/kafka/coordinator/share/ShareCoordinatorService.java#L233
> in the `shareGroupStateTopicConfigs` we ignore the 
> `share.coordinator.state.topic.min.isr`



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


Re: [PR] KAFKA-18803: The acls would appear at the wrong level of the metadata shell "tree" [kafka]

2025-02-16 Thread via GitHub


chia7712 commented on PR #18916:
URL: https://github.com/apache/kafka/pull/18916#issuecomment-2661601777

   cherry-pick to 4.0, 3.9 and 3.8


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



[jira] [Assigned] (KAFKA-18809) share.coordinator.state.topic.min.isr is not applied

2025-02-16 Thread Chia-Ping Tsai (Jira)


 [ 
https://issues.apache.org/jira/browse/KAFKA-18809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chia-Ping Tsai reassigned KAFKA-18809:
--

Assignee: Sushant Mahajan  (was: Chia-Ping Tsai)

> share.coordinator.state.topic.min.isr is not applied 
> -
>
> Key: KAFKA-18809
> URL: https://issues.apache.org/jira/browse/KAFKA-18809
> Project: Kafka
>  Issue Type: Bug
>Reporter: Chia-Ping Tsai
>Assignee: Sushant Mahajan
>Priority: Blocker
> Fix For: 4.1.0
>
>
> https://github.com/apache/kafka/blob/e330f0bf2570a27811fa20a2f446b101a7a656f3/share-coordinator/src/main/java/org/apache/kafka/coordinator/share/ShareCoordinatorService.java#L233
> in the `shareGroupStateTopicConfigs` we ignore the 
> `share.coordinator.state.topic.min.isr`



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[PR] KAFKA-18733: Updating share group record acks metric (2/N) [kafka]

2025-02-16 Thread via GitHub


apoorvmittal10 opened a new pull request, #18924:
URL: https://github.com/apache/kafka/pull/18924

   The PR corrects the record acks rate for number of records acknowledged. 
Also corrects the behaviour when multiple acks type exists in a single 
acknowledgement batch.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



Re: [PR] KAFKA-18806: Resource leak in MemoryRecordsBuilder in RecordsUtil [kafka]

2025-02-16 Thread via GitHub


chia7712 commented on code in PR #18919:
URL: https://github.com/apache/kafka/pull/18919#discussion_r1957408912


##
clients/src/main/java/org/apache/kafka/common/record/RecordsUtil.java:
##
@@ -89,10 +89,11 @@ protected static ConvertedRecords 
downConvert(Iterablehttps://github.com/apache/kafka/blob/d0e516a87207b04f6266c59712775b5f01c1ed87/clients/src/main/java/org/apache/kafka/common/record/RecordsUtil.java#L124
   
   Additionally, `RecordsUtil` is useless in trunk and it will be removed by 
#18897



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



Re: [PR] MINOR: Remove unused member in DynamicBrokerConfig [kafka]

2025-02-16 Thread via GitHub


chia7712 commented on PR #18915:
URL: https://github.com/apache/kafka/pull/18915#issuecomment-2661609606

   the failed test is traced by 
https://issues.apache.org/jira/browse/KAFKA-15474


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



Re: [PR] KAFKA-18806: Resource leak in MemoryRecordsBuilder in RecordsUtil [kafka]

2025-02-16 Thread via GitHub


xelathan commented on code in PR #18919:
URL: https://github.com/apache/kafka/pull/18919#discussion_r1957411153


##
clients/src/main/java/org/apache/kafka/common/record/RecordsUtil.java:
##
@@ -89,10 +89,11 @@ protected static ConvertedRecords 
downConvert(Iterablehttps://github.com/apache/kafka/pull/18897) then closing wouldn't 
really be necessary. I'll go ahead and close the PR.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



Re: [PR] KAFKA-18806: Resource leak in MemoryRecordsBuilder in RecordsUtil [kafka]

2025-02-16 Thread via GitHub


xelathan closed pull request #18919: KAFKA-18806: Resource leak in 
MemoryRecordsBuilder in RecordsUtil
URL: https://github.com/apache/kafka/pull/18919


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



Re: [PR] MINOR: Remove unused member in DynamicBrokerConfig [kafka]

2025-02-16 Thread via GitHub


chia7712 merged PR #18915:
URL: https://github.com/apache/kafka/pull/18915


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



[jira] [Resolved] (KAFKA-18755) Align timeout in kafka-share-groups.sh with other group-related tools

2025-02-16 Thread Chia-Ping Tsai (Jira)


 [ 
https://issues.apache.org/jira/browse/KAFKA-18755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chia-Ping Tsai resolved KAFKA-18755.

Resolution: Fixed

> Align timeout in kafka-share-groups.sh with other group-related tools
> -
>
> Key: KAFKA-18755
> URL: https://issues.apache.org/jira/browse/KAFKA-18755
> Project: Kafka
>  Issue Type: Sub-task
>  Components: tools
>Affects Versions: 4.1.0
>Reporter: Andrew Schofield
>Assignee: Jimmy Wang
>Priority: Major
> Fix For: 4.1.0
>
>
> kafka-share-groups.sh uses a timeout of 5000ms for listing groups. 
> kafka-groups.sh uses 3ms. As a result, we have seen situations in testing 
> where kafka-share-groups.sh times out, when kafka-groups.sh does not. There 
> is no good reason for the difference so we should align them.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


Re: [PR] KAFKA-18806: Resource leak in MemoryRecordsBuilder in RecordsUtil [kafka]

2025-02-16 Thread via GitHub


chia7712 commented on code in PR #18919:
URL: https://github.com/apache/kafka/pull/18919#discussion_r1957411650


##
clients/src/main/java/org/apache/kafka/common/record/RecordsUtil.java:
##
@@ -89,10 +89,11 @@ protected static ConvertedRecords 
downConvert(Iterable I'll go ahead and close the PR.
   
   yes and thanks for your contribution. Please feel free to ping me if you 
have other PRs needs to be reviewed.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



[jira] [Resolved] (KAFKA-18806) Resource leak in MemoryRecordsBuilder in RecordsUtil

2025-02-16 Thread Chia-Ping Tsai (Jira)


 [ 
https://issues.apache.org/jira/browse/KAFKA-18806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chia-Ping Tsai resolved KAFKA-18806.

Resolution: Won't Fix

see https://github.com/apache/kafka/pull/18919#discussion_r1957408912

> Resource leak in MemoryRecordsBuilder in RecordsUtil
> 
>
> Key: KAFKA-18806
> URL: https://issues.apache.org/jira/browse/KAFKA-18806
> Project: Kafka
>  Issue Type: Bug
>Reporter: Luke Chen
>Assignee: Alex Tran
>Priority: Major
>  Labels: newbie
>
> We forgot to close the MemoryRecordsBuilder resource (which contains streams) 
> here: 
> https://github.com/apache/kafka/blob/e330f0bf2570a27811fa20a2f446b101a7a656f3/clients/src/main/java/org/apache/kafka/common/record/RecordsUtil.java#L92.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


Re: [PR] KAFKA-18755 Align timeout in kafka-share-groups.sh [kafka]

2025-02-16 Thread via GitHub


chia7712 merged PR #18908:
URL: https://github.com/apache/kafka/pull/18908


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org