> On Oct. 16, 2018, 2:28 a.m., Jason Dere wrote: > > ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java > > Lines 4761 (patched) > > <https://reviews.apache.org/r/68946/diff/3/?file=2098216#file2098216line5019> > > > > Should this be on by default? If there are a lot of external tables > > (especially on s3), the metastore could be spending a lot of time doing > > auto discover. Could also affect the running of other MetastoreTaskThreads. > > Prasanth_J wrote: > Yeah. I think this should be default. This will remove manual msck step > or periodic msck query (via cron job). This thread kicks in once every 5 > minutes but if the previous attempt is not done yet it will skip an attempt > so as to avoid queue'ing up of background tasks. Also it will use high batch > size by default so that in most case there should be 1 MS request per table. > MSCK thread also runs in a thread pool. The only place this background thread > could be blocked is when exclusive lock is obtained on a table (which gets > released after txn timeout of 300s). > We could probably restrict this only for EXTERNAL table types (currently > it defaults to both EXTERNAL and MANAGED). Since managed is ACID by default > we can avoid scanning managed tables. Changing ACID table layout out of band > is shooting themselves in the foot anyway.
Actually, thinking about it again. I think I will leave the table types (metastore.partition.management.table.types) as such. Required for partition retention. If user adds retention period to managed table, they don't have to do anything. Added a fix in new patch that won't acquire lock unless it is required (change detected, adding or dropping partition). - Prasanth_J ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/68946/#review209582 ----------------------------------------------------------- On Oct. 16, 2018, 12:21 a.m., Prasanth_J wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/68946/ > ----------------------------------------------------------- > > (Updated Oct. 16, 2018, 12:21 a.m.) > > > Review request for hive, Ashutosh Chauhan and Jason Dere. > > > Bugs: HIVE-20707 > https://issues.apache.org/jira/browse/HIVE-20707 > > > Repository: hive-git > > > Description > ------- > > HIVE-20707: Automatic partition management > > > Diffs > ----- > > common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 92a1c31 > hbase-handler/src/test/results/positive/external_table_ppd.q.out edcbe7e > hbase-handler/src/test/results/positive/hbase_binary_storage_queries.q.out > 1209c88 > hbase-handler/src/test/results/positive/hbase_ddl.q.out ccd4148 > hbase-handler/src/test/results/positive/hbase_queries.q.out eeb97f0 > hbase-handler/src/test/results/positive/hbasestats.q.out 5a4aea9 > > itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java > a9d7468 > ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 807f159 > ql/src/java/org/apache/hadoop/hive/ql/exec/ExplainTask.java 46bf088 > ql/src/java/org/apache/hadoop/hive/ql/metadata/CheckResult.java 0b4240f > ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveMetaStoreChecker.java > 598bb2e > ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java cff32d3 > ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java > 29f6ecf > ql/src/java/org/apache/hadoop/hive/ql/plan/CreateTableDesc.java 27f677e > > ql/src/test/org/apache/hadoop/hive/ql/exec/TestMsckCreatePartitionsInBatches.java > ce2b186 > > ql/src/test/org/apache/hadoop/hive/ql/exec/TestMsckDropPartitionsInBatches.java > 9480d38 > > ql/src/test/org/apache/hadoop/hive/ql/metadata/TestHiveMetaStoreChecker.java > a2a0583 > ql/src/test/queries/clientpositive/msck_repair_acid.q PRE-CREATION > ql/src/test/queries/clientpositive/partition_discovery.q PRE-CREATION > ql/src/test/results/clientpositive/create_like.q.out f4a5ed5 > ql/src/test/results/clientpositive/create_like_view.q.out 870f280 > ql/src/test/results/clientpositive/default_file_format.q.out 0adf5ae > ql/src/test/results/clientpositive/druid_topn.q.out 179902a > ql/src/test/results/clientpositive/explain_locks.q.out ed7f1e8 > ql/src/test/results/clientpositive/llap/external_table_purge.q.out 24c778e > ql/src/test/results/clientpositive/llap/mm_exim.q.out ee6cf06 > ql/src/test/results/clientpositive/llap/strict_managed_tables2.q.out > f3b6152 > ql/src/test/results/clientpositive/llap/whroot_external1.q.out cac158c > ql/src/test/results/clientpositive/msck_repair_acid.q.out PRE-CREATION > ql/src/test/results/clientpositive/msck_repair_drop.q.out 2456734 > ql/src/test/results/clientpositive/partition_discovery.q.out PRE-CREATION > ql/src/test/results/clientpositive/rename_external_partition_location.q.out > 02cd814 > ql/src/test/results/clientpositive/repl_2_exim_basic.q.out b2bcd51 > ql/src/test/results/clientpositive/show_create_table_alter.q.out 2c75c36 > ql/src/test/results/clientpositive/show_create_table_partitioned.q.out > e554a18 > ql/src/test/results/clientpositive/show_create_table_serde.q.out 8b95c9b > ql/src/test/results/clientpositive/spark/stats_noscan_2.q.out 2d713a8 > ql/src/test/results/clientpositive/stats_noscan_2.q.out 182820f > ql/src/test/results/clientpositive/temp_table_display_colstats_tbllvl.q.out > 2a442b4 > > standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/CheckResult.java > PRE-CREATION > > standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/Warehouse.java > 294dfb7 > > standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/api/MetastoreException.java > PRE-CREATION > > standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java > 7b01678 > > standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreUtils.java > 16f4a50 > > standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreChecker.java > PRE-CREATION > > standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/Msck.java > PRE-CREATION > > standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MsckInfo.java > PRE-CREATION > > standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MsckPartitionExpressionProxy.java > PRE-CREATION > > standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java > 66977d7 > > standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/PartitionIterable.java > PRE-CREATION > > standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/PartitionManagementTask.java > PRE-CREATION > > standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreServerUtils.java > f3b3866 > > standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/RetryUtilities.java > PRE-CREATION > > standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/NonCatCallsWithCatalog.java > f750ca2 > > standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestCatalogOldClient.java > fc996c8 > > standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestPartitionManagement.java > PRE-CREATION > > standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/client/TestGetTableMeta.java > 59daa52 > > > Diff: https://reviews.apache.org/r/68946/diff/3/ > > > Testing > ------- > > > Thanks, > > Prasanth_J > >
