Wataru Yukawa created HIVE-13374:
------------------------------------
Summary: HiveServer2 hangs up if query to scan too many partition
is submitted
Key: HIVE-13374
URL: https://issues.apache.org/jira/browse/HIVE-13374
Project: Hive
Issue Type: Bug
Components: HiveServer2
Affects Versions: 1.2.1
Environment: HDP2.4.0
JDK1.8
Reporter: Wataru Yukawa
Assignee: Vaibhav Gumashta
If query to scan too many partition(for example, more than 40,000 partitions)
is submitted, hiveserver2 hangs up
hiveserver2 java heap status is the following.
https://gyazo.com/e0a35bd628a0dc904bc24f19cdb2261e
https://gyazo.com/65f12cd3875330995508ae11afd394a2
OutOfMemoryError doesn't occur. but other queries fails.
Maybe it is necessary to spill to disk...
hiveserver2.log
{code}
....
2016-03-28 13:44:28,923 INFO [HiveServer2-Handler-Pool: Thread-766923]:
io.CombineHiveInputFormat (CombineHiveInputFormat.java:getSplits(517)) - Total
number of paths: 47409, launching 50 threads to check non-combinable ones.
2016-03-28 13:44:29,444 INFO [HiveServer2-Handler-Pool: Thread-766923]:
io.CombineHiveInputFormat (CombineHiveInputFormat.java:getCombineSplits(439)) -
CombineHiveInputSplit creating pool for hdfs://.../aaa=.../yyyymmdd=20150721;
using filter path hdfs://.../aaa=.../yyyymmdd=20150721
....
2016-03-28 13:44:34,811 INFO [HiveServer2-Handler-Pool: Thread-766923]:
io.CombineHiveInputFormat (CombineHiveInputFormat.java:getCombineSplits(444)) -
CombineHiveInputSplit: pool is already created for
hdfs://.../aaa=.../yyyymmdd=20160326; using filter path
hdfs://.../aaa=.../yyyymmdd=20160326
2016-03-28 13:44:34,811 INFO [HiveServer2-Handler-Pool: Thread-766923]:
io.CombineHiveInputFormat (CombineHiveInputFormat.java:getCombineSplits(444)) -
CombineHiveInputSplit: pool is already created for
hdfs://.../aaa=.../yyyymmdd=20160327; using filter path
hdfs://.../aaa=.../yyyymmdd=20160327
2016-03-28 13:44:34,811 INFO [HiveServer2-Handler-Pool: Thread-766923]:
io.CombineHiveInputFormat (CombineHiveInputFormat.java:getCombineSplits(444)) -
CombineHiveInputSplit: pool is already created for
hdfs://.../aaa=.../yyyymmdd=20160328; using filter path
hdfs://.../aaa=.../yyyymmdd=20160328
2016-03-28 13:45:00,285 INFO
[org.apache.hadoop.util.JvmPauseMonitor$Monitor@6f5d0190]: util.JvmPauseMonitor
(JvmPauseMonitor.java:run(195)) - Detected pause in JVM or host machine (eg
GC): pause of approximately 4587ms
GC pool 'PS MarkSweep' had collection(s): count=1 time=4699ms
GC pool 'PS Scavenge' had collection(s): count=1 time=63ms
....
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
