Thank you very much ! @Long Chao 2019-04-12
lk_hadoop 发件人：Long Chao <chao.long0...@gmail.com> 发送时间：2019-04-12 18:17 主题：Re: Re: 答复: can't pass step Build Cube In-Mem 收件人："dev"<dev@kylin.apache.org> 抄送： Hi lk, I have fixed this issue, and the code is in Kylin's master branch now. If your situation is very urgent, you can apply the commit[ https://github.com/apache/kylin/commit/ed266aa98d8524a344469b1e1ead8bfd462702d8] and build a new binary package. Btw, to keep the previous behavior(optimize job using inmem algorithm as default), I just add a new configuration parameter( *kylin.cube.algorithm.inmem-auto-optimize*) to remove the above limitation, you need to set *kylin.cube.algorithm.inmem-auto-optimize *to *false*, and then the optimize job will use the algorithm as you configured(like: *kylin.cube.algorithm=layer*). On Thu, Apr 11, 2019 at 6:00 PM lk_hadoop <lk_had...@163.com> wrote: > thank you ~ @Long Chao > > 2019-04-11 > > lk_hadoop > > > > 发件人：Long Chao <chao.long0...@gmail.com> > 发送时间：2019-04-11 17:56 > 主题：Re: 答复: can't pass step Build Cube In-Mem > 收件人："dev"<dev@kylin.apache.org> > 抄送： > > Hi lk, > Optimize job will only build the newly generated cuboids in the > recommended cuboid list, usually the amount of them is not too large. > So, by default, we use inmem algorithm to build those new cuboids, > but > now the algorithm can't be overwritten by properties file. > > And I create a jira for this problem to make the algorithm > configurable. https://issues.apache.org/jira/browse/KYLIN-3950 > > On Thu, Apr 11, 2019 at 5:49 PM lk_hadoop <lk_had...@163.com> wrote: > > > I think that's not too much : > > > > Cuboid Distribution > > Current Cuboid Distribution > > [Cuboid Count: 49] [Row Count: 1117994636] > > > > Recommend Cuboid Distribution > > [Cuboid Count: 168] [Row Count: 464893216] > > > > > > 2019-04-11 > > > > lk_hadoop > > > > > > > > 发件人：Na Zhai <na.z...@kyligence.io> > > 发送时间：2019-04-11 17:42 > > 主题：答复: can't pass step Build Cube In-Mem > > 收件人："dev@kylin.apache.org"<dev@kylin.apache.org> > > 抄送： > > > > Hi, lk_hadoop. > > > > > > > > Does Cube planner recommend too many cuboid? If so, it may cause OOM. > > > > > > > > > > > > 发送自 Windows 10 版邮件<https://go.microsoft.com/fwlink/?LinkId=550986>应用 > > > > > > > > ________________________________ > > 发件人: lk_hadoop <lk_had...@163.com> > > 发送时间: Tuesday, April 9, 2019 9:21:59 AM > > 收件人: dev > > 主题: can't pass step Build Cube In-Mem > > > > hi,all : > > I'm using kylin-2.6.1-cdh57, and the source row count is 500 > million，I > > can success build cube . > > but when I use the cube planner , it has one step : Build Cube In-Mem > > for job :OPTIMIZE CUBE > > the config about the kylin_job_conf_inmem.xml is : > > > > <property> > > <name>mapreduce.map.memory.mb</name> > > <value>9216</value> > > <description></description> > > </property> > > > > <property> > > <name>mapreduce.map.java.opts</name> > > <value>-Xmx8192m -XX:OnOutOfMemoryError='kill -9 %p'</value> > > <description></description> > > </property> > > > > <property> > > <name>mapreduce.job.is-mem-hungry</name> > > <value>true</value> > > </property> > > > > <property> > > <name>mapreduce.job.split.metainfo.maxsize</name> > > <value>-1</value> > > <description>The maximum permissible size of the split metainfo > > file. > > The JobTracker won't attempt to read split metainfo files > > bigger than > > the configured value. No limits if set to -1. > > </description> > > </property> > > > > <property> > > <name>mapreduce.job.max.split.locations</name> > > <value>2000</value> > > <description>No description</description> > > </property> > > > > <property> > > <name>mapreduce.task.io.sort.mb</name> > > <value>200</value> > > <description></description> > > </property> > > > > > > finally the map job will be killed for OnOutOfMemoryError , but > when > > I giev more mem for map job , I will get another error : > > java.nio.BufferOverflowException > > > > why kylin will run the job inmem ? how can I avoid it ? > > > > > > > > 2019-04-08 > > > > > > lk_hadoop
