Thanks David for the detailed review. Appreciated. I will try to answer your most important question: what is the problem I am trying to solve here.
This KIP will surely resolve some technical problems. But that's not the original purpose. I am trying to resolve a problem in Kafka's design. The replicas serve two major function: accepting write and serve read. But if we observe a replica, we can see that most (not all, I must say) of the read and write happened only on the end of the replica. This part is complicated and need to handle with care. The majority part of the replica are immutable and just serve as a data store (most of the time). But when we manage the replica we manage it as a single piece. Like when we want to move a replica to a new broker we need to move all the data in the replica although most of the case we might just interested with some data at the end. What I am proposing is really to provide an capability to separate the concern between the data we mostly interested and also complicated to manage, with the data we know that are stable and immutable and very easy to manage. I see this is something missing from kafka's design. If we have this in the first place we don't need tiered storage, as adding more brokers / disks will easily hold more data. And we don't need a feature to read from followers as there are no leader in inactive chunks. Reading from any of them are the same. I believe this proposal will be an enabler for many many smart ideas that will make Kafka more flexible and more powerful. Hope this answers your question. Thanks again for your review! On 2 January 2025 16:53:04 GMT, David Arthur <mum...@gmail.com> wrote: >Hey De Gao, thanks for the KIP! > >As you’re probably aware, a Partition is a logical construct in Kafka. A >broker hosts a partition which is composed of physical log segments. Only >the active segment is being written to and the others are immutable. The >concept of a Chunk sounds quite similar to our log segments. > >From what I can tell reading the KIP, the main difference is that a Chunk >can have its own assignment and therefore be replicated across different >brokers. > >> Horizontal scalability: the data was distributed more evenly to brokers >in cluster. Also achieving a more flexible resource allocation. > >I think this is only true in cases where we have a small number of >partitions with a large amount of data. I have certainly seen cases where a >small number of partitions can cause trouble with balancing the cluster. > >The idea of shuffling around older data in order to spread out the load is >interesting. It does seem like it would increase the complexity of the >client a bit when it comes to consuming the old data. Usually the client >can just read from a single replica from the beginning of the log to the >end. With this proposal, the client would need to hop around between >replicas as it crossed the chunk boundaries. > >> Better load balancing: The read of partition data, especially early data >can be distributed to more nodes other than just leader nodes. > >As you know, this is already possible with KIP-392. I guess the idea with >the chunks is that clients would be reading older data from less busy >brokers (i.e., brokers which are not the leader, or perhaps not even a >follower of the active chunk). I’m not sure this would always result in >better load balancing. It seems a bit situational. > >> Increased fault tolerance: failure of leader node will not impact read >older data. > >I don’t think this proposal changes the fault tolerance. A failure of a >leader results in a failover to a follower. If a client is consuming using >KIP-392, a leader failure will not affect the consumption (besides updating >the clients metadata). > >-- > >I guess I'm missing a key point here. What problem is this trying to solve? >Is it a solution for the "single partition" problem? (i.e., a topic with >one partition and a lot of data) > >Thanks! >David A > >On Tue, Dec 31, 2024 at 3:24 PM De Gao <d...@live.co.uk> wrote: > >> Thanks for the comments. I have updated the proposal to compare with >> tiered storage and fetch from replica. Please check. >> >> Thanks. >> >> On 11 December 2024 08:51:43 GMT, David Jacot <dja...@confluent.io.INVALID> >> wrote: >> >Hi, >> > >> >Thanks for the KIP. The community is pretty busy with the Apache Kafka 4.0 >> >release so I suppose that no one really had the time to engage in >> reviewing >> >the KIP yet. Sorry for this! >> > >> >I just read the motivation section. I think that it is an interesting >> idea. >> >However, I wonder if this is still needed now that we have tier storage in >> >place. One of the big selling points of tier storage was that clusters >> >don't have to replicate tiered data anymore. Could you perhaps extend the >> >motivation of the KIP to include tier storage in the reflexion? >> > >> >Best, >> >David >> > >> >On Tue, Dec 10, 2024 at 10:46 PM De Gao <d...@live.co.uk> wrote: >> > >> >> Hi All: >> >> >> >> There were no discussion in the past week. Just want to double check if >> I >> >> missed anything? >> >> What should be the expectations on KIP discussion? >> >> >> >> Thank you! >> >> >> >> De Gao >> >> >> >> On 1 December 2024 19:36:37 GMT, De Gao <d...@live.co.uk> wrote: >> >> >Hi All: >> >> > >> >> >I would like to start the discussion of KIP-1114 Introducing Chunk in >> >> Partition. >> >> > >> >> > >> >> >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1114%3A+Introducing+Chunk+in+Partition >> >> >This KIP is complicated so I expect discussion will take longer time. >> >> > >> >> >Thank you in advance. >> >> > >> >> >De Gao >> >> >> > >
