[
https://issues.apache.org/jira/browse/HIVE-24318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
libo updated HIVE-24318:
------------------------
Attachment: HIVE-24318.patch
> When GlobalLimit is efficient, query will run twice with "Retry query with a
> different approach..."
> ---------------------------------------------------------------------------------------------------
>
> Key: HIVE-24318
> URL: https://issues.apache.org/jira/browse/HIVE-24318
> Project: Hive
> Issue Type: Bug
> Components: Hive
> Affects Versions: 2.0.1
> Environment: Hadoop 2.6.0
> Hive-2.0.1
> Reporter: libo
> Assignee: libo
> Priority: Minor
> Attachments: HIVE-24318.patch
>
>
> hive.limit.optimize.enable=true
> hive.limit.row.max.size=10000000
> hive.limit.optimize.fetch.max=1000
> hive.fetch.task.conversion.threshold=25600000000
> hive.fetch.task.conversion=more
>
> *sql eg:*
> select db_name,concat(tb_name,'test') tbname from (select * from test1.t3
> where dt='0909' limit 10)t1;
> (only partitioned table has this problem)
> *console information:*
> *......*
> Kill Command = /appcom/hadoop/bin/hadoop job -kill job_1600683831691_837491
> Hadoop job information for Stage-1: number of {color:#FF0000}mappers:
> 1{color}; number of reducers: 1
> map = 0%, reduce = 0%
> map = 100%, reduce = 0%, Cumulative CPU 6.33 sec
> map = 100%, reduce = 100%, Cumulative CPU 13.69 sec
> MapReduce Total cumulative CPU time: 13 seconds 690 msec
> Ended Job = job_1600683831691_837491
> MapReduce Jobs Launched:
> Stage-Stage-1: Map: 1 Reduce: 1 Cumulative CPU: 13.69 sec HDFS Read: 43833339
> HDFS Write: 4115 SUCCESS
> Total MapReduce CPU Time Spent: 13 seconds 690 msec
> OK
> db_name tbname
> ......
> Retry query with a different approach...
> ......
> Kill Command = /appcom/hadoop/bin/hadoop job -kill job_1600683831691_837520
> Hadoop job information for Stage-1: number of {color:#FF0000}mappers:
> 176{color}; number of reducers: 1
> ......
> as we can see, the mr run twice，first time the global limit is efficient and
> the second time is not
> *exception stack:*
> org.apache.hadoop.hive.ql.CommandNeedRetryException
> at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:147)
> at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:2022)
> at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:317)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:232)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:475)
> at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:855)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:794)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:721)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:323)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:236)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
