> On Oct. 16, 2018, 2:28 a.m., Jason Dere wrote: > > ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java > > Lines 4761 (patched) > > <https://reviews.apache.org/r/68946/diff/3/?file=2098216#file2098216line5019> > > > > Should this be on by default? If there are a lot of external tables > > (especially on s3), the metastore could be spending a lot of time doing > > auto discover. Could also affect the running of other MetastoreTaskThreads.
Yeah. I think this should be default. This will remove manual msck step or periodic msck query (via cron job). This thread kicks in once every 5 minutes but if the previous attempt is not done yet it will skip an attempt so as to avoid queue'ing up of background tasks. Also it will use high batch size by default so that in most case there should be 1 MS request per table. MSCK thread also runs in a thread pool. The only place this background thread could be blocked is when exclusive lock is obtained on a table (which gets released after txn timeout of 300s). We could probably restrict this only for EXTERNAL table types (currently it defaults to both EXTERNAL and MANAGED). Since managed is ACID by default we can avoid scanning managed tables. Changing ACID table layout out of band is shooting themselves in the foot anyway. > On Oct. 16, 2018, 2:28 a.m., Jason Dere wrote: > > ql/src/test/results/clientpositive/msck_repair_drop.q.out > > Line 127 (original), 127 (patched) > > <https://reviews.apache.org/r/68946/diff/3/?file=2098238#file2098238line127> > > > > What is the new ordering of these messages? Looks like it could be > > potential issue when diffing golden files? I think the ordering got changed as part of https://issues.apache.org/jira/browse/HIVE-20696 I will check again to see if we are using hashset anywhere else. The ordering is based on partition name so prepending catalog name "hive" might have changed the order although not sure (either hashset issue or unstable sort). > On Oct. 16, 2018, 2:28 a.m., Jason Dere wrote: > > standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/PartitionManagementTask.java > > Lines 141 (patched) > > <https://reviews.apache.org/r/68946/diff/3/?file=2098259#file2098259line141> > > > > Is this variable used? It's logged, but I think retentionSeconds should > > be used instead. good catch. will fix it. > On Oct. 16, 2018, 2:28 a.m., Jason Dere wrote: > > standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/PartitionManagementTask.java > > Lines 142 (patched) > > <https://reviews.apache.org/r/68946/diff/3/?file=2098259#file2098259line142> > > > > Might want to check for exception from TimeValidator.validate() in > > getRententionPeriodInSeconds, or else a bad setting in one table can fail > > here and prevent this from running for any tables. > > But if you do skip that table, make sure the countdown latch is updated > > appropriately. make sense. will fix it. - Prasanth_J ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/68946/#review209582 ----------------------------------------------------------- On Oct. 16, 2018, 12:21 a.m., Prasanth_J wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/68946/ > ----------------------------------------------------------- > > (Updated Oct. 16, 2018, 12:21 a.m.) > > > Review request for hive, Ashutosh Chauhan and Jason Dere. > > > Bugs: HIVE-20707 > https://issues.apache.org/jira/browse/HIVE-20707 > > > Repository: hive-git > > > Description > ------- > > HIVE-20707: Automatic partition management > > > Diffs > ----- > > common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 92a1c31 > hbase-handler/src/test/results/positive/external_table_ppd.q.out edcbe7e > hbase-handler/src/test/results/positive/hbase_binary_storage_queries.q.out > 1209c88 > hbase-handler/src/test/results/positive/hbase_ddl.q.out ccd4148 > hbase-handler/src/test/results/positive/hbase_queries.q.out eeb97f0 > hbase-handler/src/test/results/positive/hbasestats.q.out 5a4aea9 > > itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java > a9d7468 > ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 807f159 > ql/src/java/org/apache/hadoop/hive/ql/exec/ExplainTask.java 46bf088 > ql/src/java/org/apache/hadoop/hive/ql/metadata/CheckResult.java 0b4240f > ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveMetaStoreChecker.java > 598bb2e > ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java cff32d3 > ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java > 29f6ecf > ql/src/java/org/apache/hadoop/hive/ql/plan/CreateTableDesc.java 27f677e > > ql/src/test/org/apache/hadoop/hive/ql/exec/TestMsckCreatePartitionsInBatches.java > ce2b186 > > ql/src/test/org/apache/hadoop/hive/ql/exec/TestMsckDropPartitionsInBatches.java > 9480d38 > > ql/src/test/org/apache/hadoop/hive/ql/metadata/TestHiveMetaStoreChecker.java > a2a0583 > ql/src/test/queries/clientpositive/msck_repair_acid.q PRE-CREATION > ql/src/test/queries/clientpositive/partition_discovery.q PRE-CREATION > ql/src/test/results/clientpositive/create_like.q.out f4a5ed5 > ql/src/test/results/clientpositive/create_like_view.q.out 870f280 > ql/src/test/results/clientpositive/default_file_format.q.out 0adf5ae > ql/src/test/results/clientpositive/druid_topn.q.out 179902a > ql/src/test/results/clientpositive/explain_locks.q.out ed7f1e8 > ql/src/test/results/clientpositive/llap/external_table_purge.q.out 24c778e > ql/src/test/results/clientpositive/llap/mm_exim.q.out ee6cf06 > ql/src/test/results/clientpositive/llap/strict_managed_tables2.q.out > f3b6152 > ql/src/test/results/clientpositive/llap/whroot_external1.q.out cac158c > ql/src/test/results/clientpositive/msck_repair_acid.q.out PRE-CREATION > ql/src/test/results/clientpositive/msck_repair_drop.q.out 2456734 > ql/src/test/results/clientpositive/partition_discovery.q.out PRE-CREATION > ql/src/test/results/clientpositive/rename_external_partition_location.q.out > 02cd814 > ql/src/test/results/clientpositive/repl_2_exim_basic.q.out b2bcd51 > ql/src/test/results/clientpositive/show_create_table_alter.q.out 2c75c36 > ql/src/test/results/clientpositive/show_create_table_partitioned.q.out > e554a18 > ql/src/test/results/clientpositive/show_create_table_serde.q.out 8b95c9b > ql/src/test/results/clientpositive/spark/stats_noscan_2.q.out 2d713a8 > ql/src/test/results/clientpositive/stats_noscan_2.q.out 182820f > ql/src/test/results/clientpositive/temp_table_display_colstats_tbllvl.q.out > 2a442b4 > > standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/CheckResult.java > PRE-CREATION > > standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/Warehouse.java > 294dfb7 > > standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/api/MetastoreException.java > PRE-CREATION > > standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java > 7b01678 > > standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreUtils.java > 16f4a50 > > standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreChecker.java > PRE-CREATION > > standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/Msck.java > PRE-CREATION > > standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MsckInfo.java > PRE-CREATION > > standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MsckPartitionExpressionProxy.java > PRE-CREATION > > standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java > 66977d7 > > standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/PartitionIterable.java > PRE-CREATION > > standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/PartitionManagementTask.java > PRE-CREATION > > standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreServerUtils.java > f3b3866 > > standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/RetryUtilities.java > PRE-CREATION > > standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/NonCatCallsWithCatalog.java > f750ca2 > > standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestCatalogOldClient.java > fc996c8 > > standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestPartitionManagement.java > PRE-CREATION > > standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/client/TestGetTableMeta.java > 59daa52 > > > Diff: https://reviews.apache.org/r/68946/diff/3/ > > > Testing > ------- > > > Thanks, > > Prasanth_J > >
