Hi, Thanks for the KIP. As Dong said, this is not straightforward. One option would be to keep the existing system, which is very simple and rely on KIP-113 to keep the data balanced across disks. If we don't think that's good enough, we should explain why it's not in the KIP.
Thanks, Ismael On Thu, Aug 3, 2017 at 2:24 AM, Dong Lin <lindon...@gmail.com> wrote: > Hey Hu, > > Actually, the solution suggested in the KIP would not work. A broker may > receive LeaderAndIsrRequest to create a large number of partitions when > user rebalance the load across brokers. All these partitions will be > created in the same log directory if the broker selects the log directory > with the most free space. The problem is that this may cause load imbalance > later as the size of those newly-created replicas will increase quickly. > > One possible solution is to estimate the size of the replica based on the > average size of replicas in the broker. A more advanced and fine-granular > solution could estimate the expected free space based on the replica > retention configuration, the time a replica has been created, and the > current size of the replica. Can you think about it and maybe write down > the algorithm in more detail? And it may be worth thinking whether we can > simply replace the existing log directory selection algorithm without > adding a new config. > > Thanks, > Dong > > On Wed, Aug 2, 2017 at 6:10 PM, Dong Lin <lindon...@gmail.com> wrote: > > > Hey Xu, > > > > Thanks for the KIP. This is a very good idea to select log directory > based > > on the free disk space. Do you think we can simply simply change the > > implementation to select log directory based on the free disk space > instead > > of adding a new config? Or is there any good reason that user will want > to > > select log directory with the least partition number instead of the one > > with the most free disk space? > > > > Thanks, > > Dong > > > > > > On Wed, Aug 2, 2017 at 6:03 PM, Hu Xi <huxi...@hotmail.com> wrote: > > > >> Hi all, how do you think of this KIP? Any comments are welcomed. > >> > >> > >> ________________________________ > >> 发件人: Hu Xi <huxi...@hotmail.com> > >> 发送时间: 2017年7月18日 15:21 > >> 收件人: dev@kafka.apache.org > >> 主题: [DISCUSS] KIP-178: Size-based log directory selection strategy > >> > >> > >> Hi all, > >> > >> KIP-178 is created for a discussion on how LogManager selects log > >> directory. In this KIP, a new strategy is introduced to allow for the > real > >> disk spaces for each directories. Be free to drop your comments here. > >> Thanks. > >> > > > > >
