Re: [DISCUSS] KIP-1183: Unified Shared Storage

2025-05-15 Thread Xinyu Zhou
Hi Colin, Thank you for taking the time to read this KIP, and no worries, negative feedback is a catalyst for improvement. Sorry for the inappropriate description in the Motivation section; my background influenced my writing, but I didn’t mean it. I will remove it. Thanks for the reminder. I co

Re: [DISCUSS] KIP-1183: Unified Shared Storage

2025-05-15 Thread Colin McCabe
Hi Xinyu Zhou, Thanks for the KIP. It's good to see more people contributing to the community. I think this is your first KIP, so please forgive me for giving some negative feedback. KIPs need to be written in a vendor-neutral manner, for the whole community. So please do not do things like be

Re: [DISCUSS] KIP-1183: Unified Shared Storage

2025-05-14 Thread Xinyu Zhou
Hi Satish, Thanks for your incisive comments! For SD-1: I am now confident that we have a better chance of reaching a consensus on the abstract layer. I will complete this part and address all the questions from you and Luke. Once finished, I will ping you again. For SD-2: In this KIP, a partiti

Re: [DISCUSS] KIP-1183: Unified Shared Storage

2025-05-14 Thread Satish Duggana
Thanks Xinyu for accepting our request to initiate a KIP on AutoMq's proposal. SD-1: It is a good start towards building log storage layer abstractions, including LogSegment and Log, for both local and shared storage. However, more clarity is needed on how these abstractions are defined. The class

Re: [DISCUSS] KIP-1183: Unified Shared Storage

2025-05-14 Thread Xinyu Zhou
Hi Luke, Thank you for your review and insightful questions. 1. Regarding the future plan for this KIP, once we establish an abstract log layer, we might implement a default version on object storage for Kafka. For other types of shared storage, we'll decide based on community needs. 2. In my vi

Re: [DISCUSS] KIP-1183: Unified Shared Storage

2025-05-14 Thread Luke Chen
Hi Xinyu, Thanks for the proposal! I agree abstracting the log and logSegment layers can allow users to implement any kinds of storage mediums. Some high-level comments: 1. I expect to see the future plan of this KIP, but can't find it. What do you expect to achieve the "shared storage" goal afte

[DISCUSS] KIP-1183: Unified Shared Storage

2025-05-13 Thread Xinyu Zhou
Dear Kafka Community, I am proposing a new KIP to introduce a unified shared storage solution for Kafka, aiming to enhance its scalability and flexibility. This KIP is inspired by the ongoing discussions around KIP-1150 and KIP-1176, which explore leveraging object storage to achieve cost and elas