Hi ,
I am trying to run query using stats with following flags as set,but it is 
always running map reduce job instead of getting direct result from metastore 
(Hive 0.13.0.2.1.2.1-471).can some please suggest me to run optimized query or 
a workaround it .
Thanks in advance

Set hive.stats.reliable=true
Set hive.fetch.column.stats=true
Set hive.stats.fetch.partition.stats=true
Set hive.compute.query.using.stats=true
query
explain select max(load_inst_id) from ndnslog_orc1 where query_date > 
'2012-01-01';//I have also tried explain select max(load_inst_id) from 
ndnslog_orc1 where query_date  = '2013-01-01' but same result as I have only 
one partition .
OK
STAGE DEPENDENCIES:
  Stage-1 is a root stage
  Stage-0 is a root stage

STAGE PLANS:
  Stage: Stage-1
    Map Reduce
      Map Operator Tree:
          TableScan
            alias: ndnslog_orc1
            filterExpr: (query_date > '2012-01-01') (type: boolean)
            Statistics: Num rows: 331575 Data size: 2652601 Basic stats: 
COMPLETE Column stats: NONE
            Select Operator
              expressions: load_inst_id (type: bigint)
              outputColumnNames: load_inst_id
              Statistics: Num rows: 331575 Data size: 2652601 Basic stats: 
COMPLETE Column stats: NONE
              Group By Operator
                aggregations: max(load_inst_id)
                mode: hash
                outputColumnNames: _col0
                Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE 
Column stats: NONE
                Reduce Output Operator
                  sort order:
                  Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE 
Column stats: NONE
                  value expressions: _col0 (type: bigint)
      Reduce Operator Tree:
        Group By Operator
          aggregations: max(VALUE._col0)
          mode: mergepartial
          outputColumnNames: _col0
          Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column 
stats: NONE
          Select Operator
            expressions: _col0 (type: bigint)
            outputColumnNames: _col0
            Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column 
stats: NONE
            File Output Operator
              compressed: false
              Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column 
stats: NONE
              table:
                  input format: org.apache.hadoop.mapred.TextInputFormat
                  output format: 
org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
                  serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe

  Stage: Stage-0
    Fetch Operator
      limit: -1


where I had already run analyze statement as I can see statistics in hive meta 
store.

My insert statement is

set hive.stats.autogather=false;
set hive.fetch.column.stats=true;
set hive.stats.fetch.partition.stats=true;
set hive.compute.query.using.stats=true;
set hive.cbo.enable=true;
--set mapred.reduce.tasks=20;
--set hive.exec.reducers.max=100;
--set mapreduce.job.reduce.slowstart.completedmaps=0.95;
--set mapreduce.task.io.sort.mb=1600;
set mapred.job.queue.name=queue1;

use ${database};
--use ssa;

insert into table ${managed_table} partition (query_date)
select
${hivevar:LOADID} as load_inst_id,
src_filename,
from_unixtime(query_time) as query_time,
client_ip,
query_type,
query_domain,
response_domain,
ttl,
response_type,
response_info,
to_date(from_unixtime(query_time)) as query_date
from  ${external_table} ;

/Navdeep

Reply via email to