[
https://issues.apache.org/jira/browse/HIVE-15803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15852245#comment-15852245
]
Rajesh Balamohan commented on HIVE-15803:
-----------------------------------------
[~hagleitn] - It is not slow, it gets stuck.
Another option is to use CachedThreadPool (which would spin up additional
threads on need basis) instead of FixedThreadPool in executor service. But
again, if there are 50000 partitions with nesting, it would end up throwing
"unable to create native thread". Passing "null" (instead of pool)
https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveMetaStoreChecker.java#L499
would ensure that only certain amount of threads are there doing the
processing and any further nesting has to be handled in the same thread
context. That would still improve the perf compared to no-thread pool scenario
as 15 threads would be processing instead of 1.
> msck can hang when nested partitions are present
> ------------------------------------------------
>
> Key: HIVE-15803
> URL: https://issues.apache.org/jira/browse/HIVE-15803
> Project: Hive
> Issue Type: Bug
> Components: Metastore
> Reporter: Rajesh Balamohan
> Assignee: Rajesh Balamohan
> Priority: Minor
>
> Steps to reproduce.
> {noformat}
> CREATE TABLE `repairtable`( `col` string) PARTITIONED BY ( `p1` string,
> `p2` string)
> hive> dfs -mkdir -p /apps/hive/warehouse/test.db/repairtable/p1=c/p2=a/p3=b;
> hive> dfs -touchz
> /apps/hive/warehouse/test.db/repairtable/p1=c/p2=a/p3=b/datafile;
> hive> set hive.mv.files.thread;
> hive.mv.files.thread=15
> hive> set hive.mv.files.thread=1;
> hive> MSCK TABLE repairtable;
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
