Hi All: I have updated the KIP to be more specific on the motivation based on the comments.Please review as you can. Appreciated. If no more review to follow I will submit the KIP for vote. Thank you!
On 3 January 2025 22:36:06 GMT, De Gao <d...@live.co.uk> wrote: >Thanks for the review. >This is an interesting idea. Indeed this will significantly reduce the data >need to be copied. But this may need to take the TTL time to get the new >replica join the ISRs. Also we need to consider how to handle the partitions >that will only do purge by data size. > >On 2 January 2025 18:11:49 GMT, Kamal Chandraprakash ><kamal.chandraprak...@gmail.com> wrote: >>Hi Deo, >> >>Thanks for the KIP! >> >>"However the limit of messages in a single partition replica is very big. >>This could lead to very big partitions (~TBs). Moving those partitions are >>very time consuming and have a big impact on system performance." >> >>One way to do faster rebalance is to have a latest-offset replica build >>strategy when expanding the replicas for a partition >>and ensure that the expanded replica does not serve as a leader until the >>data in the older nodes expires by retention time/size. >>Currently, Kafka supports only the earliest-offset strategy during >>reassignment. And, this strategy will only work for topics >>with cleanup policy set to "delete". >> >>-- >>Kamal >> >>On Thu, Jan 2, 2025 at 10:23 PM David Arthur <mum...@gmail.com> wrote: >> >>> Hey De Gao, thanks for the KIP! >>> >>> As you’re probably aware, a Partition is a logical construct in Kafka. A >>> broker hosts a partition which is composed of physical log segments. Only >>> the active segment is being written to and the others are immutable. The >>> concept of a Chunk sounds quite similar to our log segments. >>> >>> From what I can tell reading the KIP, the main difference is that a Chunk >>> can have its own assignment and therefore be replicated across different >>> brokers. >>> >>> > Horizontal scalability: the data was distributed more evenly to brokers >>> in cluster. Also achieving a more flexible resource allocation. >>> >>> I think this is only true in cases where we have a small number of >>> partitions with a large amount of data. I have certainly seen cases where a >>> small number of partitions can cause trouble with balancing the cluster. >>> >>> The idea of shuffling around older data in order to spread out the load is >>> interesting. It does seem like it would increase the complexity of the >>> client a bit when it comes to consuming the old data. Usually the client >>> can just read from a single replica from the beginning of the log to the >>> end. With this proposal, the client would need to hop around between >>> replicas as it crossed the chunk boundaries. >>> >>> > Better load balancing: The read of partition data, especially early data >>> can be distributed to more nodes other than just leader nodes. >>> >>> As you know, this is already possible with KIP-392. I guess the idea with >>> the chunks is that clients would be reading older data from less busy >>> brokers (i.e., brokers which are not the leader, or perhaps not even a >>> follower of the active chunk). I’m not sure this would always result in >>> better load balancing. It seems a bit situational. >>> >>> > Increased fault tolerance: failure of leader node will not impact read >>> older data. >>> >>> I don’t think this proposal changes the fault tolerance. A failure of a >>> leader results in a failover to a follower. If a client is consuming using >>> KIP-392, a leader failure will not affect the consumption (besides updating >>> the clients metadata). >>> >>> -- >>> >>> I guess I'm missing a key point here. What problem is this trying to solve? >>> Is it a solution for the "single partition" problem? (i.e., a topic with >>> one partition and a lot of data) >>> >>> Thanks! >>> David A >>> >>> On Tue, Dec 31, 2024 at 3:24 PM De Gao <d...@live.co.uk> wrote: >>> >>> > Thanks for the comments. I have updated the proposal to compare with >>> > tiered storage and fetch from replica. Please check. >>> > >>> > Thanks. >>> > >>> > On 11 December 2024 08:51:43 GMT, David Jacot >>> <dja...@confluent.io.INVALID> >>> > wrote: >>> > >Hi, >>> > > >>> > >Thanks for the KIP. The community is pretty busy with the Apache Kafka >>> 4.0 >>> > >release so I suppose that no one really had the time to engage in >>> > reviewing >>> > >the KIP yet. Sorry for this! >>> > > >>> > >I just read the motivation section. I think that it is an interesting >>> > idea. >>> > >However, I wonder if this is still needed now that we have tier storage >>> in >>> > >place. One of the big selling points of tier storage was that clusters >>> > >don't have to replicate tiered data anymore. Could you perhaps extend >>> the >>> > >motivation of the KIP to include tier storage in the reflexion? >>> > > >>> > >Best, >>> > >David >>> > > >>> > >On Tue, Dec 10, 2024 at 10:46 PM De Gao <d...@live.co.uk> wrote: >>> > > >>> > >> Hi All: >>> > >> >>> > >> There were no discussion in the past week. Just want to double check >>> if >>> > I >>> > >> missed anything? >>> > >> What should be the expectations on KIP discussion? >>> > >> >>> > >> Thank you! >>> > >> >>> > >> De Gao >>> > >> >>> > >> On 1 December 2024 19:36:37 GMT, De Gao <d...@live.co.uk> wrote: >>> > >> >Hi All: >>> > >> > >>> > >> >I would like to start the discussion of KIP-1114 Introducing Chunk in >>> > >> Partition. >>> > >> > >>> > >> > >>> > >> >>> > >>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1114%3A+Introducing+Chunk+in+Partition >>> > >> >This KIP is complicated so I expect discussion will take longer time. >>> > >> > >>> > >> >Thank you in advance. >>> > >> > >>> > >> >De Gao >>> > >> >>> > >>> >>> >>> -- >>> David Arthur >>>
