Re: Out of Memory while generating ORC Splits

2017-09-13 Thread Rajesh Balamohan
>> Also, this parameter is set to false by default should this be set to true ? This is set to false for predictable query performance with multiple runs. If set to true, there is a chance of footer getting evicted during GC and get into unpredictable performance. >> I see that the number of mappe

Re: Out of Memory while generating ORC Splits

2017-09-13 Thread Jay
Thanks Rajesh, this helped solve the OOM issue. I was going through the wiki documentation for this parameter and was not able to understand it clearly, can you please explain the significance of this ? I have 2 questions:- 1. Also, this parameter is set to false by default should this be set to t

Re: Out of Memory while generating ORC Splits

2017-09-13 Thread Rajesh Balamohan
With "HYBRID" can you try with "hive.orc.cache.use.soft.references=true"? That should help in preventing OOM with Hybrid strategy. ~Rajesh.B On Wed, Sep 13, 2017 at 2:54 PM, Jay wrote: > Hi All, > > I am running a simple select query as below > > select distinct vehicle_no from > rmd.gets_dw_e

Out of Memory while generating ORC Splits

2017-09-13 Thread Jay
Hi All, I am running a simple select query as below select distinct vehicle_no from rmd.gets_dw_eoa_eng_rec_dtl_orc_ext_concat_final_eng3 where incident_dt = '2999-01-01'; The table is a 2 level partitioned table as shown below drwx-- - gpadmin hdfs 0 2017-09-12 14:36 /apps/hive/