A more problem is confusing me now, I read this blog http://kylin.apache.org/blog/2017/02/23/by-layer-spark-cubing/ <http://kylin.apache.org/blog/2017/02/23/by-layer-spark-cubing/> ,and got the key point of by-layer spark cubing is "a RDD per layer”, and during the cubing, I take a look at Spark DAG, is that a spark job for a layer ?
my cube has 11 dimensions, the first spark job has 2 stages, Is the stage 1 get the first layer( 11-D Cuboid) RDD(in your blog is RDD1) ? In second job, it has 3 stages, the first stage is skipped , Are the remained 2 stages get the second layer(10-D cuboids) RDD (RDD2) ? and In every follow spark jobs, always skipped some former stages ,just leave last 2 stages to implements. Please forgive my ignorance of this. Thanks. > 在 2017年3月3日，下午6:10，ShaoFeng Shi <[email protected]> 写道： > > I'm afraid not that simple; the embedded spark in Kylin is v1.6.3, and the > Kylin code is compiled with 1.6.3 API; if you want to switch to 2.1, need > recompile and re-package. That is doable, you can take a try. > > Get Outlook for iOS <https://aka.ms/o0ukef> > > > > On Thu, Mar 2, 2017 at 1:37 PM +0700, "Mars Xu" <[email protected] > <mailto:[email protected]>> wrote: > > Hi, Shaofeng, > > Can I change the spark version to spark 2.1 using > “kylin.engine.spark-conf” to config ? > >> 在 2017年3月1日，下午5:27，ShaoFeng Shi <[email protected] >> <mailto:[email protected]>> 写道： >> >> Kylin embed a spark, which is spark v1.6.3; I added a blog to briefly >> introduce this, later will add a doc to instruct on how to config: >> >> https://kylin.apache.org/blog/2017/02/23/by-layer-spark-cubing/ >> <https://kylin.apache.org/blog/2017/02/23/by-layer-spark-cubing/> >> >> To customize the spark conf, please do that in kylin.properties with prefix >> "kylin.engine.spark-conf." >> >> 2017-03-01 15:49 GMT+08:00 Mars Xu <[email protected] >> <mailto:[email protected]>>: >> Hi , >> I have install kylin 2.0 beta on my cluster which has already deploy a >> spark 2.1, but when I startup kylin server and build cube, the Web UI of >> Cube Build Step show it is spark 1.6.3. >> >> Is that kylin already include a spark version ? or some other else I need >> to configure ? >> >> >> Thanks >> >> Mars. >> >> >> >> -- >> Best regards, >> >> Shaofeng Shi 史少锋 >> >
