Hey Hu, I am not sure it is OK. Say kafka-reassign-partitions.sh is used to move 100 replicas to a broker. The the controller will send LeaderAndIsrRequest asking this broker to be the follower of these 100 partitions. While it is true that the broker will create replicas sequentially, but they will be created in a very short period of time (e.g. 2 seconds) and thus the replicas will be put in the same log directory that has the most free space at the time this broker receives the LeaderAndIsrRequest. Do you think this is a problem?
Dong On Thu, Aug 3, 2017 at 7:36 PM, Hu Xi <huxi...@hotmail.com> wrote: > Hi Dong, some thoughts on your second mail. Since currently logs for > multiple partitions are created sequentially not in parallel, it's probably > okay for us to simply select the directory with most disk spaces in a > single round of `nextLogDir` calling. which can be guaranteed to lead to > extreme skew. Does it make any senses? > > > ________________________________ > 发件人: Hu Xi <huxi...@hotmail.com> > 发送时间: 2017年8月3日 16:51 > 收件人: dev@kafka.apache.org > 主题: 答复: 答复: [DISCUSS] KIP-178: Size-based log directory selection strategy > > > Dong, yes, many thanks for the comments from the second mail. Will take > some time to figure out an algorithm to better handle the situation you > mentioned. Thanks again. > > > ________________________________ > 发件人: Dong Lin <lindon...@gmail.com> > 发送时间: 2017年8月3日 12:07 > 收件人: dev@kafka.apache.org > 主题: Re: 答复: [DISCUSS] KIP-178: Size-based log directory selection strategy > > Hu, I think this is worth discussion even if it doesn't require new config. > Could you also read my second email? > > On Wed, Aug 2, 2017 at 6:17 PM, Hu Xi <huxi...@hotmail.com> wrote: > > > Thanks Dong, do you mean it is more like a naive improvement and no KIP > > is needed then? > > > > ________________________________ > > 发件人: Dong Lin <lindon...@gmail.com> > > 发送时间: 2017年8月3日 9:10 > > 收件人: dev@kafka.apache.org > > 主题: Re: [DISCUSS] KIP-178: Size-based log directory selection strategy > > > > Hey Xu, > > > > Thanks for the KIP. This is a very good idea to select log directory > based > > on the free disk space. Do you think we can simply simply change the > > implementation to select log directory based on the free disk space > instead > > of adding a new config? Or is there any good reason that user will want > to > > select log directory with the least partition number instead of the one > > with the most free disk space? > > > > Thanks, > > Dong > > > > > > On Wed, Aug 2, 2017 at 6:03 PM, Hu Xi <huxi...@hotmail.com> wrote: > > > > > Hi all, how do you think of this KIP? Any comments are welcomed. > > > > > > > > > ________________________________ > > > 发件人: Hu Xi <huxi...@hotmail.com> > > > 发送时间: 2017年7月18日 15:21 > > > 收件人: dev@kafka.apache.org > > > 主题: [DISCUSS] KIP-178: Size-based log directory selection strategy > > > > > > > > > Hi all, > > > > > > KIP-178 is created for a discussion on how LogManager selects log > > > directory. In this KIP, a new strategy is introduced to allow for the > > real > > > disk spaces for each directories. Be free to drop your comments here. > > > Thanks. > > > > > >
