[jira] [Resolved] (KAFKA-16718) Add AdminClient.deleteShareGroupOffsets

2025-04-30 Thread Andrew Schofield (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-16718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Schofield resolved KAFKA-16718. -- Fix Version/s: 4.1.0 Resolution: Fixed > Add AdminClient.deleteShareGroupOffset

[jira] [Created] (KAFKA-19218) Add missing leader epoch to read share group state summary response

2025-04-30 Thread Andrew Schofield (Jira)
Andrew Schofield created KAFKA-19218: Summary: Add missing leader epoch to read share group state summary response Key: KAFKA-19218 URL: https://issues.apache.org/jira/browse/KAFKA-19218 Project:

Re: [DISCUSS] KIP-1140: Avoid to return null value in Map from public api of consumer

2025-04-30 Thread Chia-Ping Tsai
hi all, >Also, it would be useful to think through the behavior of ListOffsetsResult.partitionResult(final TopicPartition partition) too. It seems there are three options: 1. Keep the current behavior - it's inconsistent, as other result classes, such as DeleteRecordsResult, don't have a

[jira] [Created] (KAFKA-19220) add tests to ensure the internal configs don't return by public APIs by default

2025-04-30 Thread Chia-Ping Tsai (Jira)
Chia-Ping Tsai created KAFKA-19220: -- Summary: add tests to ensure the internal configs don't return by public APIs by default Key: KAFKA-19220 URL: https://issues.apache.org/jira/browse/KAFKA-19220 P

Re: [DISCUSS] KIP-1159: Large message reference based Serializer

2025-04-30 Thread Omnia Ibrahim
Hi Luke, >> 3. What does "LargeMessageFormatter" do in the process? >> I thought all we want to do is to replace the "large value data" into a >> path, and consumers will read the path via blob store class. >> All these should be done in serializer/deserializer, so why do we need the >> formatter?

Re: [DISCUSS] KIP-1166: Improve high-watermark replication

2025-04-30 Thread José Armando García Sancio
Hi Colin, On Mon, Apr 28, 2025 at 8:43 PM Colin McCabe wrote: > Maybe I missed something, but even after reading the discussion below, I > still don't understand the rationale for separating the "RPC version too old" > and "high watermark not known" cases. Is the idea that separating these case

Re: [DISCUSS] KIP-1166: Improve high-watermark replication

2025-04-30 Thread José Armando García Sancio
Hi Alyssa, On Tue, Apr 29, 2025 at 1:51 PM Alyssa Huang wrote: > Wondering if there is perhaps a typo - under *KRaft Handling *the wording > mentions parking the request if the HWM in the request is >= to the leader's You are correct and a good catch. There were some inconsistencies in the predi

Re: [DISCUSS] KIP-1140: Avoid to return null value in Map from public api of consumer

2025-04-30 Thread Chia-Ping Tsai
hi Ken, there is another inconsistent case for the offset-related APIs. see https://github.com/apache/kafka/pull/19578#discussion_r2068913749 Best, Chia-Ping Chia-Ping Tsai 於 2025年4月30日 週三 下午7:03寫道: > hi all, > > >Also, it would be useful to think through the behavior > of ListOffsetsResult.pa

[jira] [Created] (KAFKA-19221) IOException on log segment close shouldn't be ignored

2025-04-30 Thread Gaurav Narula (Jira)
Gaurav Narula created KAFKA-19221: - Summary: IOException on log segment close shouldn't be ignored Key: KAFKA-19221 URL: https://issues.apache.org/jira/browse/KAFKA-19221 Project: Kafka Issue

Re: [DISCUSS] KIP-1166: Improve high-watermark replication

2025-04-30 Thread Alyssa Huang
Thanks José for the edits, just one more clarification on the wording of this section The current implementation parks the FETCH request if all of these > conditions are true: fetch request has a wait time, fetch request has a > wait time, fetch request requires data, fetch request doesn't have en

RE: [DISCUSS] KIP-1121: Compression acceleration in Kafka

2025-04-30 Thread Denloye, Olasoji
Hello all, I would like to call a vote soon so if anyone wants to contribute to the discussion or bring up other concerns, you are welcome to comment. Thanks, Olasoji -Original Message- From: Denloye, Olasoji Sent: Wednesday, April 23, 2025 10:22 AM To: dev@kafka.apache.org Subject: R

Re: [DISCUSS] KIP-1162: Redesign ClientQuotaCallback#updateClusterMetadata

2025-04-30 Thread Chia-Ping Tsai
hi Kuan-Po > In TopicImage, there is a similar method, Map partitions(), but we can't directly reuse its return value here. As you mentioned, we'll still need to iterate through all partitions to build the collection. That said, since this KIP proposes to make PartitionRegistration implements Part

Re: [DISCUSS] KIP-1121: Compression acceleration in Kafka

2025-04-30 Thread Greg Harris
Hi Olasoji, Thanks for updating the KIP. 2. I'm still not satisfied with the description of the public API in the KIP. It mentions these classes: * org.apache.kafka.common.utils.ByteBufferOutputStream * org.apache.kafka.common.utils.ByteBufferInputStream * org.apache.kafka.common.utils.BufferSupp

Re: [DISCUSS] KIP-1166: Improve high-watermark replication

2025-04-30 Thread José Armando García Sancio
Hi Alyssa, On Wed, Apr 30, 2025 at 12:52 PM Alyssa Huang wrote: > Is the intent to say that in addition to all the current conditions, > the HWM check is an additional condition that *must *be met in order for > the fetch request to be parked? Just confirming since I'm not sure if I > should inte

[jira] [Created] (KAFKA-19223) Explicit HWM replication

2025-04-30 Thread Jira
José Armando García Sancio created KAFKA-19223: -- Summary: Explicit HWM replication Key: KAFKA-19223 URL: https://issues.apache.org/jira/browse/KAFKA-19223 Project: Kafka Issue Ty

[jira] [Created] (KAFKA-19224) Faster HWM replication for ISR topic partitions

2025-04-30 Thread Jira
José Armando García Sancio created KAFKA-19224: -- Summary: Faster HWM replication for ISR topic partitions Key: KAFKA-19224 URL: https://issues.apache.org/jira/browse/KAFKA-19224 Project: K

Re: [DISCUSS] KIP-1162: Redesign ClientQuotaCallback#updateClusterMetadata

2025-04-30 Thread Colin McCabe
Hi Kuan Po Tseng, Yes, the old interface extending the new one could work. I don't have a strong preference about this, as long as someone with a correctly configured and written implementation of the NEW ClientQuotaCallback interface can upgrade from 4.x to 5.0 without hitting compatibility pr

Re: [DISCUSS] KIP-1140: Avoid to return null value in Map from public api of consumer

2025-04-30 Thread Chia-Ping Tsai
hi Jun > It seems that consumer.committed() has the same behavior. If you pass in a non-existing partition, it returns a map with a null value. So, if we want to make a change, it would be useful to make it consistent too. yes, both consumers have same behavior. For another, `listStreamsGroupOff

Re: [DISCUSS] KIP-1166: Improve high-watermark replication

2025-04-30 Thread Colin McCabe
Hi José, Thanks for the explanation. That makes sense to me. That was the biggest quesiton I had about the KIP, I am +1 about the rest of it. Do you have a PR for this yet? It will be interesting to see the latency improvements. regards, Colin On Wed, Apr 30, 2025, at 08:31, José Armando Gar

Re: [DISCUSS] KIP-1166: Improve high-watermark replication

2025-04-30 Thread Jun Rao
Hi, Jose, Thanks for updating the KIP. A few more comments. JR5. The motivation section still sounds like this is a KRaft only problem. It would be useful to cover the issue with fetch from followers too. JR6. Since there is an inter broker RPC change, we need to document the upgrade path with a

[jira] [Created] (KAFKA-19222) Invalid FENCED_MEMBER_EPOCH error to ConsumerGroupHeartbeat

2025-04-30 Thread Travis Bischel (Jira)
Travis Bischel created KAFKA-19222: -- Summary: Invalid FENCED_MEMBER_EPOCH error to ConsumerGroupHeartbeat Key: KAFKA-19222 URL: https://issues.apache.org/jira/browse/KAFKA-19222 Project: Kafka

Re: [DISCUSS] KIP-1140: Avoid to return null value in Map from public api of consumer

2025-04-30 Thread Jun Rao
Hi, Chia-Ping, It seems that consumer.committed() has the same behavior. If you pass in a non-existing partition, it returns a map with a null value. So, if we want to make a change, it would be useful to make it consistent too. Thanks, Jun On Wed, Apr 30, 2025 at 9:05 AM Chia-Ping Tsai wrote:

Re: [DISCUSS] KIP-1161: cleanup.policy shouldn't be empty

2025-04-30 Thread Jun Rao
Hi, Jiunn-Yang, Thanks for the reply. Do we really need isEmptyAllowed? It's awkward since the method name is inNonEmpty. Also, it's not clear when it's set to true. Thanks, Jun On Fri, Apr 25, 2025 at 6:26 AM 黃竣陽 wrote: > Hi Jun, > > Thank you for pointing that out. > > I have revised the K

[jira] [Created] (KAFKA-19216) Eliminate flakiness in kafka.server.share.SharePartitionTest

2025-04-30 Thread Andrew Schofield (Jira)
Andrew Schofield created KAFKA-19216: Summary: Eliminate flakiness in kafka.server.share.SharePartitionTest Key: KAFKA-19216 URL: https://issues.apache.org/jira/browse/KAFKA-19216 Project: Kafka

[jira] [Created] (KAFKA-19219) improve KRaft migration log

2025-04-30 Thread Luke Chen (Jira)
Luke Chen created KAFKA-19219: - Summary: improve KRaft migration log Key: KAFKA-19219 URL: https://issues.apache.org/jira/browse/KAFKA-19219 Project: Kafka Issue Type: Improvement Affects Ver

[jira] [Created] (KAFKA-19217) ShareConsumerTest.testComplexShareConsumer is unreliable

2025-04-30 Thread Andrew Schofield (Jira)
Andrew Schofield created KAFKA-19217: Summary: ShareConsumerTest.testComplexShareConsumer is unreliable Key: KAFKA-19217 URL: https://issues.apache.org/jira/browse/KAFKA-19217 Project: Kafka

RE: Re: [DISCUSS] KIP-1117 Support keystore with multiple alias entries

2025-04-30 Thread Rahul Nirgude
Hi All, Gentle reminder Please review the KIP and the PR. KIP : https://cwiki.apache.org/confluence/display/KAFKA/KIP-1117%3A+Support+keystore+with+multiple+alias+entries PR : https://github.com/apache/kafka/pull/17560 Also, I would like to start the voting process so any feedback would be ap

[DISCUSS] KIP-1176: Tiered Storage for Active Log Segment

2025-04-30 Thread Henry Haiying Cai
Link to the KIP:  https://cwiki.apache.org/confluence/display/KAFKA/KIP-1176%3A+Tiered+Storage+for+Active+Log+Segment Motivation In KIP-405, the community has proposed and implemented the tiered storage for old Kafka log segment files, when the log segments is older than  local.retention.ms, it be

[jira] [Created] (KAFKA-19225) Tiered Storage Support for Active Log Segment

2025-04-30 Thread Henry Cai (Jira)
Henry Cai created KAFKA-19225: - Summary: Tiered Storage Support for Active Log Segment Key: KAFKA-19225 URL: https://issues.apache.org/jira/browse/KAFKA-19225 Project: Kafka Issue Type: New Featu

Re: [DISCUSS] KIP-1119: Add support for SSL hot reload

2025-04-30 Thread Moncef Abboud
Gaurav, Colin, Thank you for your replies. > It appears that inotify on Linux doesn't emit events for bind mounts. This is indeed a valid concern and it seems that this a well-known issue.. As Colin suggested, we can work around this by manually polling or by leveraging a library that uses polli

Re: Kafka support for Java 24+

2025-04-30 Thread Stig Rohde Døssing
Just wanted to provide a short status since it's been a bit: Luke got the last bits merged, so 3.9 is tested against Java 23. There's a PR to move trunk to testing against Java 24 at https://github.com/apache/kafka/pull/19514. It was blocked for a while waiting for Gradle to release, and now it's

Re: [DISCUSS] KIP-1166: Improve high-watermark replication

2025-04-30 Thread Jun Rao
Hi, Ismael, Thanks for the pointers. Hi, Jose, KIP-392 originally considered propagating HWM to followers. If there is a performance concern, we could focus only on the KRaft metadata topic. Also, KIP-392 implemented HWM propagation without adding the HWM field in the fetch request. Instead, the

Re: [DISCUSS] KIP-1166: Improve high-watermark replication

2025-04-30 Thread José Armando García Sancio
Hi Jun, On Wed, Apr 30, 2025 at 1:09 PM Jun Rao wrote: > JR5. The motivation section still sounds like this is a KRaft only problem. > It would be useful to cover the issue with fetch from followers too. Okay. I updated the motivation section to also talk about fetch-from-followers. I was hesita

Re: [DISCUSS] KIP-1166: Improve high-watermark replication

2025-04-30 Thread Ismael Juma
And also https://issues.apache.org/jira/browse/KAFKA-9952 Ismael On Wed, Apr 30, 2025 at 2:18 PM Ismael Juma wrote: > Hi Jose, > > We did indeed run into a perf regression related to increased fetch > request rate due to hw propagation to followers: > > https://issues.apache.org/jira/browse/KAF

Re: [DISCUSS] KIP-1166: Improve high-watermark replication

2025-04-30 Thread Ismael Juma
Hi Jose, We did indeed run into a perf regression related to increased fetch request rate due to hw propagation to followers: https://issues.apache.org/jira/browse/KAFKA-9731 Ismael On Wed, Apr 30, 2025 at 12:34 PM José Armando García Sancio wrote: > Hi Alyssa, > > On Wed, Apr 30, 2025 at 12: