The following are my notes from today's KIP meeting. The link to the video has been added to https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals .
- KIP-253 - partition expansion: We discussed a few things. (1) Is it useful to backfill a compacted topic? The main use case is to rebuild the application states. If the new partition has the existing data, rebuilding the state can be done easily by reading from a single partition. Otherwise, an application has to read both the child and the parent partition to rebuild the state. This is possible, but can be complicated. Jan will do an exercise to see how complicated this is. (2) What's the best way to add the backfilling support if we want to do it? We can do this on the server side or on the client side. The former potentially makes the coordination easier. The latter potentially reduces the memory footprint on the server for reshuffling. We need to think through how to support EOS message format and how to throttle the process to avoid overwhelming the cluster. (3) Linear hashing vs doubling partitions. It seems that Linear hashing is more general. (4) Partition splitting in Kinesis. This is done differently since it doesn't allow customized partitioning. It doesn't support compacted topic either. (5) Sticky partition assignment. It could be useful to support a partition assignment strategy where the child partition is assigned together with the parent partition to a consumer instance so that the local state doesn't have to be moved immediately. (6) Consumer callback on partition splitting. This could still be useful if the states are maintained globally. Thanks, Jun On Thu, Apr 5, 2018 at 10:31 PM, Jeff Chao <jc...@salesforce.com> wrote: > Hi Jun, same for me. jeffc...@me.com > > Thanks, > Jeff Chao > > On Thu, Apr 5, 2018 at 9:06 PM, Jun Rao <j...@confluent.io> wrote: > > > Hi, James, Jeff, > > > > Added you to the invitation. > > > > Jun > > > > On Thu, Apr 5, 2018 at 6:11 PM, Jeff Widman <j...@jeffwidman.com> wrote: > > > > > Please add me as well: j...@jeffwidman.com > > > > > > thanks > > > > > > On Thu, Apr 5, 2018 at 5:16 PM, James Cheng <wushuja...@gmail.com> > > wrote: > > > > > > > Jun, > > > > > > > > Can you add me as well? wushuja...@gmail.com <mailto: > > > wushuja...@gmail.com> > > > > > > > > Thanks, > > > > -James > > > > > > > > > On Apr 5, 2018, at 1:34 PM, Jun Rao <j...@confluent.io> wrote: > > > > > > > > > > Hi, Ted, Vahid, > > > > > > > > > > Added you to the invite. > > > > > > > > > > Thanks, > > > > > > > > > > Jun > > > > > > > > > > > > > > > On Thu, Apr 5, 2018 at 10:42 AM, Vahid S Hashemian < > > > > > vahidhashem...@us.ibm.com> wrote: > > > > > > > > > >> Hi Jun, > > > > >> > > > > >> I used to receive these invites, but didn't get this one. > > > > >> Please send me an invite. Thanks. > > > > >> > > > > >> Regards, > > > > >> --Vahid > > > > >> > > > > >> > > > > >> > > > > >> From: Jun Rao <j...@confluent.io> > > > > >> To: dev <dev@kafka.apache.org> > > > > >> Date: 04/05/2018 10:25 AM > > > > >> Subject: Kafka KIP meeting on Apr. 9 at 9:00am PDT > > > > >> > > > > >> > > > > >> > > > > >> Hi, Everyone, > > > > >> > > > > >> We plan to have a Kafka KIP meeting this coming Monday (Apr. 9) > at > > > > 9:00am > > > > >> PDT. If you plan to attend but haven't received an invite, please > > let > > > me > > > > >> know. The following is the agenda. > > > > >> > > > > >> Agenda: > > > > >> KIP-253: Partition expansion > > > > >> > > > > >> Thanks, > > > > >> > > > > >> Jun > > > > >> > > > > >> > > > > >> > > > > >> > > > > >> > > > > > > > > > > > > > > > > > -- > > > > > > *Jeff Widman* > > > jeffwidman.com <http://www.jeffwidman.com/> | 740-WIDMAN-J (943-6265) > > > <>< > > > > > >
