[DISCUSS] KIP-1138: Clean up TopologyConfig and API for supplying configs needed by the topology

à 14:11 À : dev Objet : Re: [DISCUSS] KIP-1138: Clean up TopologyConfig and API for supplying configs needed by the topology Thanks, everyone, and thanks, Sophie, for this great summary. Indeed, the KafkaStreams constructor uses a Properties object, so it might be more concise to use it. Even if

Re: [DISCUSS] KIP-1161: cleanup.policy shouldn't be empty

Hi, Jiunn-Yang, It seems there are quite a few other configs of type list (e.g. several SSL related ones). It would be useful to understand if empty lists are valid for them or not. Thanks, Jun On Tue, May 13, 2025 at 10:12 AM Jun Rao wrote: > Hi, Luke, > > Thanks for the reply. > > "My thoug

Re: [DISCUSS] KIP-1159: Large message reference based Serializer

Hi Luke, sorry for late response. > IMO, the retry should be the logic inside the > "large.message.payload.store.class" > implementation. If we really want it, I think we need to make it clear in > which circumstance we will retry. For example, if it's an unknown exception > thrown from S3 API, wh

Re: [DISCUSS] KIP-1179: Introduce remote.log.manager.follower.thread.pool.size config

Best Regards, > > > Kuan-Po > > > > > > On Mon, May 19, 2025 at 11:16 PM Kamal Chandraprakash < > > > kamal.chandraprak...@gmail.com> wrote: > > > > > > > Hi Kuan-Po, > > > > > > > > The proposal to a

Re: [DISCUSS] KIP-1179: Introduce remote.log.manager.follower.thread.pool.size config

gt; > The proposal to add the new > remote.log.manager.follower.thread.pool.size > > > config, LGTM. > > > Thanks for making this change! > > > > > > -- > > > Kamal > > > > > > On Mon, May 12, 2025 at 10:19 PM KuanPo Tseng > &

Re: [DISCUSS] KIP-1153: Extending support for Microsecond Precision for Kafka Connect

Hi Kafka Community, If there is no other feedback. I would like to start a VOTE. On Mon, May 19, 2025 at 6:51 PM pritam kumar wrote: > Hi Kafka Community, > Here is the KIP LInk -> > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=347933282 > > On Mon, May 19, 2025 at 6:47 PM pr

Re: [DISCUSS] KIP-646: Serializer API should support ByteBuffer

Hi everyone, I want to bump this thread manually. Thanks for your attention. Best, TengYao TengYao Chi 於 2025年5月6日 週二 下午12:27寫道： > Hello everyone, > > I want to start a discussion thread on KIP-646 > , which proposes enabling > the Serializer API t

Re: [DISCUSS] KIP-1179: Introduce remote.log.manager.follower.thread.pool.size config

; > -- > > Kamal > > > > On Mon, May 12, 2025 at 10:19 PM KuanPo Tseng > > wrote: > > > > > Hello everyone, > > > > > > I’d like to discuss a KIP regarding introducing a new configuration, > > > remote.log.manager.follower.thread.pool.size > > > > > > Thank you! > > > > > > KIP link: https://cwiki.apache.org/confluence/x/xAlWFQ > > > > > > Best, > > > Kuan-Po Tseng > > > > > >

Re: [DISCUSS] KIP-1179: Introduce remote.log.manager.follower.thread.pool.size config

i Kuan-Po, > > The proposal to add the new remote.log.manager.follower.thread.pool.size > config, LGTM. > Thanks for making this change! > > -- > Kamal > > On Mon, May 12, 2025 at 10:19 PM KuanPo Tseng > wrote: > > > Hello everyone, > > > > I’d lik

Re: [DISCUSS] KIP-1179: Introduce remote.log.manager.follower.thread.pool.size config

Hi Kuan-Po, The proposal to add the new remote.log.manager.follower.thread.pool.size config, LGTM. Thanks for making this change! -- Kamal On Mon, May 12, 2025 at 10:19 PM KuanPo Tseng wrote: > Hello everyone, > > I’d like to discuss a KIP regarding introducing a new conf

Re: [DISCUSS] KIP-1153: Kafka Streams `CloseOptions` should not have a public constructor

Hi all, I’d like to manually bump this thread. If there’s no further feedback, I’ll start the vote tomorrow. Best Regards, Jiunn-Yang > 黃竣陽 於 2025年5月8日 晚上8:35 寫道： > > Hi all, > > Thanks for all the feedback. I’ve updated the KIP and introduced a > CloseOptionsInternal class that provides a

Re: [DISCUSS] KIP-1173: Connect Storage Topics Sharing Across Clusters

Hi Greg / Chris, I hope I’ve addressed all the questions — please let me know if any further clarification is needed. Also, when you have a moment, could you please review the KIP and let me know if there's anything else that needs attention? Best regards, Pritam On Sat, May 3, 2025 at 10:43 PM

Re: [DISCUSS] KIP-1153: Extending support for Microsecond Precision for Kafka Connect

Hi Kafka Community, Here is the KIP LInk -> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=347933282 On Mon, May 19, 2025 at 6:47 PM pritam kumar wrote: > Hi Kafka Community, > > Could you please review this KIP proposal when you have a chance? > > Thank you, > Pritam > > > On

Re: [DISCUSS] KIP-1153: Extending support for Microsecond Precision for Kafka Connect

Hi Kafka Community, Could you please review this KIP proposal when you have a chance? Thank you, Pritam On Sat, May 3, 2025 at 10:53 PM pritam kumar wrote: > Hi Chris, > Can you please review this one too? > > On Sat, Apr 5, 2025 at 7:00 AM pritam kumar > wrote: > >> Hi Kafka Community, >> S

Re: [DISCUSS] KIP-1130: Add metrics indicating the connection count exceeds

Hi TengYao, Thanks for the KIP, I have some questions: AM1: There exists metric "kafka.network:type=Acceptor,name=AcceptorBlockedPercent,listener={listener}" which records blocked time for all the requests which are waiting for connection slot, because of throttling time or no connection slot.

Re: [DISCUSS] KIP-1159: Large message reference based Serializer

Hi Omnia, Thanks for the explanation and update. It's better now. Questions: > 2. It's not clear how the "retry count" comes into play in the KIP. It > needs more explanation. My initial thinking is the retry configs are a must for all blob stores, so we can provide them, and validate them for fr

Re: [DISCUSS] KIP-1140: Avoid to return null value in Map from public api of consumer

Hi Jun, chia, >> 1. What's the desired way for returning a map when some of the values are >> not present? Users can use the input value to identify which values are missing from the map. To me, this is more intuitive in terms of semantics. For now, it’s reasonable to keep the original design. W

Re: [DISCUSS] KIP-1150 Diskless Topics

ype of > > > topics that would make use of Object Storage as the primary source of > > > storage. However, as this KIP is big we decided to split it into > multiple > > > related KIPs. > > > We have the motivational KIP-1150 ( > > > > > > > https:/

Re: [DISCUSS] KIP-1150 Diskless Topics

e of > > topics that would make use of Object Storage as the primary source of > > storage. However, as this KIP is big we decided to split it into multiple > > related KIPs. > > We have the motivational KIP-1150 ( > > > > https://cwiki.apache.org/confluence/display/KAF

Re: [DISCUSS] KIP-1150 Diskless Topics

it it into multiple > > related KIPs. > > We have the motivational KIP-1150 ( > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1150%3A+Diskless+Topics > > ) > > that aims to discuss if Apache Kafka should aim to have this type of > > feature at a

Re: [DISCUSS] KIP-1150 Diskless Topics

f > > topics that would make use of Object Storage as the primary source of > > storage. However, as this KIP is big we decided to split it into multiple > > related KIPs. > > We have the motivational KIP-1150 ( > > > > https://cwiki.apache.org/confluence/displa

Re: [DISCUSS] KIP-1150 Diskless Topics

t Storage as the primary source of > > storage. However, as this KIP is big we decided to split it into multiple > > related KIPs. > > We have the motivational KIP-1150 ( > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1150%3A+Diskless+Topics > &g

Re: [DISCUSS] KIP-1140: Avoid to return null value in Map from public api of consumer

hi Jun > For 1, is it better to skip the missing value in the return map or include it? To me, including every item passed in from the input seems more intuitive and less error prone for the users. Yes, I agree that using null for now is suitable. Especially considering the growing number of offs

Re: [DISCUSS] KIP-1183: Unified Shared Storage

wice, is a huge burden. For KIP-1176, which I really like, it mainly tries to solve the replication traffic cost issue, but doesn't leverage other advantages of shared storage. We can certainly accept KIP-1176, but what's next? We may still need to discuss how to better support Kafka on clo

Re: [DISCUSS] KIP-1183: Unified Shared Storage

Hi Xinyu Zhou, Thanks for the KIP. It's good to see more people contributing to the community. I think this is your first KIP, so please forgive me for giving some negative feedback. KIPs need to be written in a vendor-neutral manner, for the whole community. So please do not do things like be

Re: [DISCUSS] KIP-1180: Add a generic feature level metric

Hi Kevin, On Thu, May 15, 2025 at 10:52 AM Kevin Wu wrote: > Thanks for all the comments about the metric type field for the minimum and > maximum supported feature levels. I agree they are software version > specific. Also, since they are shared across the broker and controller like > the Finali

Re: [DISCUSS] KIP-1140: Avoid to return null value in Map from public api of consumer

Hi, Chia-Ping, Yes, there are two things to consider. 1. What's the desired way for returning a map when some of the values are not present? 2. Once we agreed on a desired way, how to make that consistent across all APIs. For 1, is it better to skip the missing value in the return map or includ

Re: [DISCUSS] KIP-1174: Expose addReadOnlyStateStore in DSL (StreamsBuilder)

Hello Kafka Community, Please review this KIP. Thank you. Best regards, Siddhartha Devineni On Fri, May 2, 2025 at 4:15 PM Siddhartha Devineni < siddhartha.devin...@gmail.com> wrote: > Hello all, > > Here is the link to the PR that is implemented according to the KIP, > please review it for any

Re: [DISCUSS] KIP-1180: Add a generic feature level metric

Hi all, Thanks for all the comments about the metric type field for the minimum and maximum supported feature levels. I agree they are software version specific. Also, since they are shared across the broker and controller like the FinalizedLevel metric, having two separate metrics is redundant an

Re: [DISCUSS] KIP-1178: Introduce remote.max.partition.fetch.bytes in Consumer

returned for a partition than expected. > > Thanks, > Andrew > > > From: Kamal Chandraprakash > Sent: 08 May 2025 19:24 > To: dev@kafka.apache.org > Subject: Re: [DISCUSS] KIP-1178: Introduce > remote.max.partition.fetch.

Re: [DISCUSS] KIP-1150 Diskless Topics

ld make use of Object Storage as the primary source of > storage. However, as this KIP is big we decided to split it into multiple > related KIPs. > We have the motivational KIP-1150 ( > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1150%3A+Diskless+Topics) > that aims to discu

Re: [DISCUSS] KIP-1180: Add a generic feature level metric

Hi Kevin, Thanks for the KIP. 4. How about using a new metric type to collect all feature metrics? So we don’t need to have different metric names in controller and broker. For example: * kafka.server:type=FeatureMetrics,name=FinalizedLevel,featureName=X * kafka.server:type=FeatureMetrics,name=Mi

Re: [DISCUSS] KIP-1140: Avoid to return null value in Map from public api of consumer

hi Jun I understand the use case where the "Result" is expected to represent both "exist" and "non-exist." However, I'm concerned about consistency across our APIs. Not all of Admin APIs follow this pattern; for example, listPartitionReassignments and describeProducers do not. Designing a single A

Re: [DISCUSS] KIP-1176: Tiered Storage for Active Log Segment

IP-405, and it is kind of inherited from > > KIP-405, we should make it clear in the KIP. > > > > > > Thanks. > > Luke > > > > > > > > > > On Thu, May 8, 2025 at 9:54 AM Xinyu Zhou wrote: > > > > > Hi Henry, > > > &

Re: [DISCUSS] KIP-1140: Avoid to return null value in Map from public api of consumer

Hi, Jiunn-Yang, Thanks for the reply. If we omit the non-existing value in the return map, for applications that care about the missing value, is it convenient for them to write the code? They need to save the input to find this out, right? Jun On Fri, May 9, 2025 at 4:39 AM 黃竣陽 wrote: > Hi

Re: [DISCUSS] KIP-1183: Unified Shared Storage

Hi Satish, Thanks for your incisive comments! For SD-1: I am now confident that we have a better chance of reaching a consensus on the abstract layer. I will complete this part and address all the questions from you and Luke. Once finished, I will ping you again. For SD-2: In this KIP, a partiti

Re: [DISCUSS] KIP-1176: Tiered Storage for Active Log Segment

e: > > 1. It looks like we only move "logs" to the fast cloud storage, not the > > index files, producer snapshots,...etc. Is that right? > > Because this is different from KIP-405, and it is kind of inherited from > > KIP-405, we should make it clear in the KIP. > > &

Re: [DISCUSS] KIP-1183: Unified Shared Storage

Thanks Xinyu for accepting our request to initiate a KIP on AutoMq's proposal. SD-1: It is a good start towards building log storage layer abstractions, including LogSegment and Log, for both local and shared storage. However, more clarity is needed on how these abstractions are defined. The class

Re: [DISCUSS] KIP-1176: Tiered Storage for Active Log Segment

9:54 AM Xinyu Zhou wrote: > > > Hi Henry, > > > > Thank you for your detailed reply. The answer makes sense to me, and you're > > right, KIP-1176 has a clear and specific scope and is expected to have a > > quick path to implement it. > > > > I

Re: [DISCUSS] Apache Kafka 4.1.0 release

Hi, Reminder that today is KIP freeze. If you have a KIP that has been approved before the end of the day, you can add it to the 4.1.0 release plan. The next milestone is feature freeze on June 4. Thanks, Mickael On Tue, May 13, 2025 at 6:48 PM Mickael Maison wrote: > > Hi Kirk, > Yes it was a

Re: [DISCUSS] KIP-1183: Unified Shared Storage

Hi Luke, Thank you for your review and insightful questions. 1. Regarding the future plan for this KIP, once we establish an abstract log layer, we might implement a default version on object storage for Kafka. For other types of shared storage, we'll decide based on community needs. 2. In my vi

Re: [DISCUSS] KIP-1183: Unified Shared Storage

Hi Xinyu, Thanks for the proposal! I agree abstracting the log and logSegment layers can allow users to implement any kinds of storage mediums. Some high-level comments: 1. I expect to see the future plan of this KIP, but can't find it. What do you expect to achieve the "shared storage" goal afte

Re: [DISCUSS] Apache Kafka 3.9.1 release

Hi Jose, Please hold on cherry-picking the commit. I'm afraid the RC tag will be wrong if there's new commits. Please let us finish the build first. Thank you. Luke On Tue, May 13, 2025 at 11:15 PM José Armando García Sancio wrote: > HI TengYao and Luke, > > I would like to cherry-pick the com

Re: [DISCUSS] KIP-1150 Diskless Topics

new KIP discussion about introducing a new type of > topics that would make use of Object Storage as the primary source of > storage. However, as this KIP is big we decided to split it into multiple > related KIPs. > We have the motivational KIP-1150 ( > https://cwiki.apache.org/conflu

Re: [DISCUSS] KIP-1180: Add a generic feature level metric

Hi, Kevin, Thanks for the reply. 4. There seems to be some inconsistency. For MinimumSupportedLevel, we have two metrics, one with the package kafka.controller and another with kafka.server. For FinalizedLevel, we only have one metric with the package kafka.server. Could we choose a more consiste

Re: [DISCUSS] KIP-1178: Introduce remote.max.partition.fetch.bytes in Consumer

FetchRequest and how sensitive it is to more data being returned for a partition than expected. Thanks, Andrew From: Kamal Chandraprakash Sent: 08 May 2025 19:24 To: dev@kafka.apache.org Subject: Re: [DISCUSS] KIP-1178: Introduce remote.max.partition.fetch.by

Re: [DISCUSS] KIP-1161: cleanup.policy shouldn't be empty

Hi, Luke, Thanks for the reply. "My thought is that this behavior has been existed for years (maybe after "cleanup.policy" was introduced?), there could be users relying on the empty cleanup.policy for a long time. " If you do a google search on "kafka infinite retention", you will get links lik

Re: [DISCUSS] KIP-1150 Diskless Topics

s this KIP is big we decided to split it into multiple > related KIPs. > We have the motivational KIP-1150 ( > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1150%3A+Diskless+Topics > ) > that aims to discuss if Apache Kafka should aim to have this type of > feature at al

Re: [DISCUSS] Apache Kafka 4.1.0 release

Hi Kirk, Yes it was approved before KIP freeze, so feel free to add it to the release plan. (I would have done it now but the wiki seems to be down at the moment) Hi Jose, Ack Thanks, Mickael On Tue, May 13, 2025 at 6:17 PM José Armando García Sancio wrote: > > HI Mickael, > > I have added "KIP

Re: [DISCUSS] Apache Kafka 4.1.0 release

HI Mickael, I have added "KIP-1166: Improve high-watermark replication" to the 4.1.0 release wiki page. Thanks, -- -José

Re: [DISCUSS] KIP-1180: Add a generic feature level metric

Hi Kevin, The metrics that you added for supported versions, " kafka.controller:type=KafkaController,name=MaximumSupportedLevel,featureName=X" and "kafka.server:type=broker-metadata-metrics,name=maximum-supported-feature-level,featureName=X", the metric group name doesn't seem accurate. Those met

Re: [DISCUSS] Apache Kafka 3.9.1 release

HI TengYao and Luke, I would like to cherry-pick the commit for this issue (https://issues.apache.org/jira/browse/KAFKA-18345) to the 3.9 branch. I see that we are still in the process of releasing 3.9.1. My cherry pick doesn't need to be in 3.9.1 and it can wait until 3.9.2. Can I cherry pick the

Re: [DISCUSS] KIP-1180: Add a generic feature level metric

Hi Jun, On Mon, May 12, 2025 at 7:21 PM Jun Rao wrote: > 5. Some of the metric names are camel case and some others use dash. It > would be useful to be consistent. They are consistent within their metrics group or type in JMX. I would be in favor of a KIP that standardizes how metrics are named

Re: [DISCUSS] KIP-1180: Add a generic feature level metric

HI Kevin, On Tue, May 13, 2025 at 9:56 AM Kevin Wu wrote: > 4. Maybe I'm missing something, but I think the MetadataLoader is used by > both the broker and controller, so having the one metric works for both > node types. The CurrentMetadataVersion metric is currently reported on both > the broke

Re: [DISCUSS] KIP-1180: Add a generic feature level metric

Hey Jun, Thanks for the comments: 4. Maybe I'm missing something, but I think the MetadataLoader is used by both the broker and controller, so having the one metric works for both node types. The CurrentMetadataVersion metric is currently reported on both the broker and controller. 5. What is the

Re: [DISCUSS] KIP-1161: cleanup.policy shouldn't be empty

Hello Mickael, Thanks for this nice catch. > One weirdness of cleanup.policy is that you can set it to > "delete,delete,delete,compact,compact" for example. Setting delete or > compact multiple times has no functional impact. Currently, only the process.roles config checks for duplicate values

Re: [DISCUSS] KIP-1161: cleanup.policy shouldn't be empty

FYI while trying with an AlterConfigPolicy to stop the cleanup.policy being emptied through a AlterConfigOp.OpType.SUBTRACT, I stumbled in this behavior that doesn't allow to block an empty policy in Zookeeper versions of Kafka https://issues.apache.org/jira/browse/KAFKA-19026 On Tue, 13 May 2025

[DISCUSS] KIP-1183: Unified Shared Storage

Dear Kafka Community, I am proposing a new KIP to introduce a unified shared storage solution for Kafka, aiming to enhance its scalability and flexibility. This KIP is inspired by the ongoing discussions around KIP-1150 and KIP-1176, which explore leveraging object storage to achieve cost and elas

Re: [DISCUSS] KIP-1176: Tiered Storage for Active Log Segment

he metadata to the follower, to let > > the > > > follower download the logs from fast cloud storage to avoid cross-az > > cost. > > > 3. If the metadata local file is not found on the leader node, we can > > fall > > > back to pass the pure logs direct

Re: [DISCUSS] KIP-1161: cleanup.policy shouldn't be empty

Hi Jun, > Regarding the motivation, currently, we never documented that an empty cleanup.policy implies infinite retention. In fact, if one leaves cleanup.policy empty, it actually breaks remote storage since it will stop the cleaning based on local retention size and time too. So, leaving cleanup

Re: [DISCUSS] KIP-1161: cleanup.policy shouldn't be empty

hi all, Given Luke mentioned the existence of use cases, it is worth considering the compatibility issue. In fact, I had previously encountered this use case but advised customers to utilize retention configurations instead. > We could also consider supporting an empty cleanup.policy by fixing

Re: [DISCUSS] KIP-1180: Add a generic feature level metric

Hi, Kevin, Thanks for the updated KIP. A couple of more comments. 4. Should we expose the FinalizedLevel metric on the controller too? 5. Some of the metric names are camel case and some others use dash. It would be useful to be consistent. Jun On Mon, May 12, 2025 at 7:06 AM Kevin Wu wrote:

Re: [DISCUSS] KIP-1161: cleanup.policy shouldn't be empty

Hi, Luke, Regarding the motivation, currently, we never documented that an empty cleanup.policy implies infinite retention. In fact, if one leaves cleanup.policy empty, it actually breaks remote storage since it will stop the cleaning based on local retention size and time too. So, leaving cleanup

[DISCUSS] KIP-1179: Introduce remote.log.manager.follower.thread.pool.size config

Hello everyone, I’d like to discuss a KIP regarding introducing a new configuration, remote.log.manager.follower.thread.pool.size Thank you! KIP link: https://cwiki.apache.org/confluence/x/xAlWFQ Best, Kuan-Po Tseng

Re: [DISCUSS] KIP-1161: cleanup.policy shouldn't be empty

Hi, Thanks for the KIP. One weirdness of cleanup.policy is that you can set it to "delete,delete,delete,compact,compact" for example. Setting delete or compact multiple times has no functional impact. I don't think it's something that has to be addressed, but while you're looking at the configura

Re: [DISCUSS] KIP-1161: cleanup.policy shouldn't be empty

Hi Luke, Chia, Thanks for your information. >> FYI, there are indeed use cases that need to keep all history data without >> a cleanup policy. I agree that we should maintain backward compatibility. Therefore, I will update the KIP as follows: - If the user sets an empty cleanup.policy, we wil

Re: [DISCUSS] KIP-1180: Add a generic feature level metric

Hey Chia-Ping and Justine, Okay, that makes sense about the minimum version changing at some point. I'll add these metrics to this KIP. Thanks for the insightful discussion. Best, Kevin Wu On Fri, May 9, 2025 at 4:54 PM Kevin Wu wrote: > Hey Chia-Ping and Justine, > > Thanks for the explanatio

Re: [DISCUSS] KIP-1176: Tiered Storage for Active Log Segment

er and followers nodes. > > So > > > when the lagged follower fetches some old data in the active log > segment, > > > the leader can still respond with the metadata to the follower, to let > > the > > > follower download the logs from fast cloud storage to

Re: [DISCUSS] KIP-1161: cleanup.policy shouldn't be empty

hi Luke We can print warnings for empty cleanup.policy in 4.x and in 5.0 we throw exception directly to make it fail fast Jiunn: WDYT? Best, Chia-Ping > Luke Chen 於 2025年5月12日 上午10:52 寫道： > > Hi Jiunn-Yang, > > Thanks for the KIP. > > Comments: > 1. "it is acceptable because an empty cl

Re: [DISCUSS] KIP-1161: cleanup.policy shouldn't be empty

Hi Jiunn-Yang, Thanks for the KIP. Comments: 1. "it is acceptable because an empty cleanup.policy is considered invalid in Kafka. Therefore, this trade-off is justified." Could you explain more about this? Why is this trade-off justified? If we think the empty cleanup.policy is invalid in kafka,

Re: [DISCUSS] KIP-1148: Remove log.cleaner.enable configuration

hi TengYao please sync the update to the vote thread. thanks Best, Chia-Ping TengYao Chi 於 2025年5月12日 週一 上午12:07寫道： > Hi Chia-Ping > Nice catch! > I have updated the KIP. > > Best, > TengYao > > Chia-Ping Tsai 於 2025年5月11日 週日 下午4:34寫道： > > > hi Teng > > > > Sorry for raising issue on the acce

Re: [DISCUSS] KIP-1148: Remove log.cleaner.enable configuration

Hi Chia-Ping Nice catch! I have updated the KIP. Best, TengYao Chia-Ping Tsai 於 2025年5月11日 週日 下午4:34寫道： > hi Teng > > Sorry for raising issue on the accepted KIP. > > In 5.0 the log cleaner should be always enabled and so the config > log.cleaner.threads should NOT accept min value of zero, rig

Re: [DISCUSS] KIP-1148: Remove log.cleaner.enable configuration

hi Teng Sorry for raising issue on the accepted KIP. In 5.0 the log cleaner should be always enabled and so the config log.cleaner.threads should NOT accept min value of zero, right? Hence, could you please add the behavior changes “the min value should be 1” to this KIP? Best, Chia-Ping On

Re: [DISCUSS] KIP-1148: Remove log.cleaner.enable configuration

hi Teng Sorry for raising issue on the accepted KIP. In 5.0 the log cleaner should be always enabled and so the config log.cleaner.threads should NOT accept min value of zero, right? Hence, could you please add the behavior changes “the min value should be 1” to this KIP? Best, Chia-Ping On

Re: [DISCUSS] KIP-1100: Consider renaming org.apache.kafka.server:type=AssignmentsManager

Hi chia, File a Jira to trace it https://issues.apache.org/jira/browse/KAFKA-19262 Best Regards, Jiunn-Yang > 黃竣陽 於 2025年5月11日 下午1:41 寫道： > > Hi Ismael, chia, > > Thanks for all the feedback. I’ve updated the KIP. > >> 2. Given that we regressed in two different instances (that we found so >

Re: [DISCUSS] KIP-1100: Consider renaming org.apache.kafka.server:type=AssignmentsManager

hi Jiunn-Yang Could you please file a ticket to address Ismael comment? That can be addressed even though this KIP is still in progress. Best, Chia-Ping > 黃竣陽 於 2025年5月11日 下午1:42 寫道： > > Jiunn-Yang

Re: [DISCUSS] KIP-1100: Consider renaming org.apache.kafka.server:type=AssignmentsManager

Hi Ismael, chia, Thanks for all the feedback. I’ve updated the KIP. > 2. Given that we regressed in two different instances (that we found so > far), this indicates a test gap combined with a brittle coding pattern. We > should consider how to fix this going forward. It's not specific to the KIP

Re: [DISCUSS] KIP-1100: Consider renaming org.apache.kafka.server:type=AssignmentsManager

Hi, Thanks for the KIP. A couple of comments: 1. Nit: KIPs should not "consider" things, they propose specific and well defined changes. Given that, please update the title to match the proposal. 2. Given that we regressed in two different instances (that we found so far), this indicates a test ga

Re: [DISCUSS] KIP-1100: Consider renaming org.apache.kafka.server:type=AssignmentsManager

hi Jiunn some questions are listed below. PTAL chia_02: "at the next major release" -> could you please write the obvious version? chia_03: Could you please list the impacted metrics related to `AssignmentsManager` , such as `QueuedReplicaToDirAssignments` Best, Chia-Ping 黃竣陽 於 2025年5月7日 週

Re: [DISCUSS] KIP-1180: Add a generic feature level metric

Hi all, > so this metric's value should only change when a software upgrade/downgrade > occurs. Otherwise, the range should not change. Is that correct? Yes, Justine already gives great answers > since the minimum is always 0? As Justine’s response, we may have a feature which can’t be disabl

Re: [DISCUSS] KIP-1176: Tiered Storage for Active Log Segment

> > cross-az cost is the same. > > > > What do you think about this alternative for WAL metadata? > > > > One more question from me: > > 1. It looks like we only move "logs" to the fast cloud storage, not the > > index files, producer snapshots,...etc. Is

Re: [DISCUSS] KIP-1180: Add a generic feature level metric

Yup, this would change during a software upgrade/downgrade, which often takes longer than the feature upgrades. I think for now minimum is 0, but I do recall some folks saying that could change in the future. Justine On Fri, May 9, 2025 at 2:55 PM Kevin Wu wrote: > Hey Chia-Ping and Justine, >

Re: [DISCUSS] KIP-1180: Add a generic feature level metric

Hey Chia-Ping and Justine, Thanks for the explanation. I see where y'all are coming from, but I want to make sure I understand how the value of this metric would change. It seems to me that the supported feature range is determined by the software version, so this metric's value should only chang

Re: Re: [DISCUSS] KIP-1150 Diskless Topics

Hi Jan, Thanks for the question and interest in the KIP. I'll be participating in this discussion from the authors' side as well. KIP-E has been published today as https://cwiki.apache.org/confluence/display/KAFKA/KIP-1181%3A+Metadata+Rack+Awareness+for+Diskless+Topics and the proposal there is a

[DISCUSS] KIP-1181: Metadata Rack Awareness for Diskless Topics

Hi all! We want to start the discussion thread for KIP-1181: Metadata Rack Awareness for Diskless Topics [1], which is a sub-KIP for KIP-1150 [2]. Let's use the main KIP-1150 discuss thread [3] for high-level questions, motivation, and general direction of the feature and this threa

Re: [DISCUSS] KIP-1140: Avoid to return null value in Map from public api of consumer

Hi Jun, chia Thanks for great examples. I still believe that when a map contains a key, its value should not be null. Now that we've introduced the allow.null.offsets.entries config as a bridge between the two behaviors, I trust that most users who rely on null values will have a reasonable p

Re: [DISCUSS] KIP-1180: Add a generic feature level metric

hi All Thank you, Justine, for your explanation. You are right that my point is that it would be useful to check the supported versions for each node at runtime, especially in a hybrid cluster or during a binary upgrade. I don't have a strong reason to push this into this KIP, as the functional

Re: [DISCUSS] KIP-1176: Tiered Storage for Active Log Segment

ta topic, > > the metadata is still needed to be replicated to all replicas, so the > > cross-az cost is the same. > > > > What do you think about this alternative for WAL metadata? > > > > One more question from me: > > 1. It looks like we only move "logs

Re: [DISCUSS] KIP-1176: Tiered Storage for Active Log Segment

reply. The answer makes sense to me, and you're > right, KIP-1176 has a clear and specific scope and is expected to have a > quick path to implement it. > > I also want to discuss the metadata management of WAL log segments. Is an > internal topic necessary for managing metadata? I

Re: [DISCUSS] KIP-1180: Add a generic feature level metric

Hey Kevin, I think Chia Ping was referring to not the time when the feature upgrade is happening, but when a kafka image upgrade is happening for example. We can't upgrade the feature until all brokers support it, so it would be also helpful to see that in real time? Something like that. I don't

Re: [DISCUSS] KIP-1180: Add a generic feature level metric

Hey Chia-Ping, I hadn't considered adding the supported versions for each feature as a metric, but I'm not sure if it's helpful for monitoring the progress of an upgrade/downgrade of a feature. For example, if a node doesn't support a particular feature level we're upgrading to, we shouldn't even

Re: [DISCUSS] KIP-1176: Tiered Storage for Active Log Segment

oducer snapshots,...etc. Is that right? > Because this is different from KIP-405, and it is kind of inherited from > KIP-405, we should make it clear in the KIP. > > > Thanks. > Luke > > > > > On Thu, May 8, 2025 at 9:54 AM Xinyu Zhou wrote: > >> Hi Hen

Re: [DISCUSS] KIP-1178: Introduce remote.max.partition.fetch.bytes in Consumer

y all consumers setting the > same > value. It suggests that they shouldn't be able to provide their own values > at all. > > Thanks, > Andrew > ____ > From: Kamal Chandraprakash > Sent: 08 May 2025 12:07 > To: dev@kafka.apache.org

Re: [DISCUSS] KIP-1176: Tiered Storage for Active Log Segment

agement. On Wednesday, May 7, 2025 at 06:54:23 PM PDT, Xinyu Zhou wrote: Hi Henry, Thank you for your detailed reply. The answer makes sense to me, and you're right, KIP-1176 has a clear and specific scope and is expected to have a quick path to implement it. I also want to

Re: [DISCUSS] KIP-1175: Fix the typo `PARTITIONER_ADPATIVE_PARTITIONING_ENABLE` in ProducerConfig

s no need to deprecate the DOC constant because it's private. > > Thanks, > Andrew > > From: 鍾明諺 > Sent: 08 May 2025 09:21 > To: dev@kafka.apache.org > Subject: [DISCUSS] KIP-1175: Fix the typo > `PARTITIONER_ADPATIVE_PARTITIO

Re: [DISCUSS] KIP-1180: Add a generic feature level metric

hi Kevin thanks for this KIP. It does offer the useful metrics. One question: Have you considered adding the supported versions to the metrics? Currently, admin#describeFeatures cannot return supported versions for a specific server because we cannot configure the target server handling the RPC r

Re: [DISCUSS] KIP-1180: Add a generic feature level metric

Hey Jun, Thanks for the comments. 1. I'll update the KIP. My trunk is a bit stale. 2. Yeah, the metric should report the finalized feature level for the feature. And if it is not set, the metric will report 0. 3. I'll update the KIP with a timeline. Thanks, Kevin On Wed, May 7, 2025 at 3:10 PM K

Re: [DISCUSS] KIP-1175: Fix the typo `PARTITIONER_ADPATIVE_PARTITIONING_ENABLE` in ProducerConfig

Hi Ming-Yen, Thanks for the KIP. I'm glad to see that this is going to be cleaned up :) AS1: There's no need to deprecate the DOC constant because it's private. Thanks, Andrew From: 鍾明諺 Sent: 08 May 2025 09:21 To: dev@kafka.apache.org Su

Re: [DISCUSS] KIP-1178: Introduce remote.max.partition.fetch.bytes in Consumer

apache.org Subject: [DISCUSS] KIP-1178: Introduce remote.max.partition.fetch.bytes in Consumer Hi all, I've opened the KIP-1178 to add a new config 'remote.max.partition.fetch.bytes' in the consumer. This config allows it to read from remote storage faster. https://cwiki.apache.org

Re: [DISCUSS] KIP-1153: Kafka Streams `CloseOptions` should not have a public constructor

Hi all, Thanks for all the feedback. I’ve updated the KIP and introduced a CloseOptionsInternal class that provides a getter method. Best Regards, Jiunn-Yang > Chia-Ping Tsai 於 2025年5月8日 下午1:55 寫道： > >> One more nit: we should remove the getters. There are useless. -- In > Kafka Streams, we f

  1   2   3   4   5   6   7   8   9   10   >