Thanks Da Tong for the careful code check. But actually, both BatchCubingJobBuilder and BatchCubingJobBuilder2 will call HBaseMRSteps.createCreateHTableStep, The CreateHTableJob step will calculate the regions by split parameter.
2017-01-03 16:25 GMT+08:00 Da Tong <[email protected]>: > Hi, > > We found that in Hadoop using mapred2 with yarn, the number of HFile > created by Kylin is always 1. After some investigation, we suspect that in > engine-mr, the BatchCubingJobBuilder2 works in a different way of > BatchCubingJobBuilder. BatchCubingJobBuilder will invoke > HBaseMRSteps.addSaveCuboidToHTableSteps, > which include calculating region size. But BatchCubingJobBuilder2 invoke > HBaseMRSteps.createConvertCuboidToHfileStep directly. > I am not sure that this difference is by design or not. But what we see is > that we got a single 16GB hfile in a single region even we set > > > > > > > > > > > kylin.hbase.region.cut and Kylie.hbase.hfile.size.gb. > > > > -- > TONG, Da / 佟达 >
