Hi,
The number of reducers in 'fact distinct columns' step is calculated as:
{number of normal columns need to build dict} + {UHC columns number * UHC
reducer count} + {number of cuboid row counters} + 1
the UHC reducer number can be configured by "kylin.engine.mr.uhc-reducer-count".
You need to identify which reducer is OOM, and find log in the reducer side to
identify the responsibility of the reducer. Usually the OOM caused by column's
cardinality is too high, and use dict encoding for that column. Currently there
are two solutions for this issue:
1. use other encoding method instead of dict for ultra high cardinality column.
2. increase the memory of reducer(only help when the cardinality is not very
high...)
At 2018-05-30 15:09:03, "陈星宇" <chenxingyu...@keruyun.com> wrote:
>hi,
>when i use kylin 2.3.1 build huge cube, got error at 'Extract Fact Table
>Distinct Columns',
>reduce job fail because of Java heap space, i suspect the num of reducer is
>too less, so tried many parameter to increase reducer, but is not working, see
>my parameter as below:
>
>
>
>kylin.engine.mr.mapper-input-rows 200000
>kylin.engine.mr.min-reducer-number 26
>kylin.storage.hbase.hfile-size-gb 0.5
>kylin.storage.hbase.min-region-count 28
>kylin.storage.hbase.region-cut-gb 0.5
>kylin.query.max-return-rows 7000000
>kylin.engine.mr.uhc-reducer-count 30
>kylin.storage.hbase.max-region-count 500
>
>
>
>is any suggestion for this?
>
>
>thanks
>
>
>chenxingyu
