> On April 20, 2018, 1:20 a.m., Ashutosh Chauhan wrote: > > ql/src/java/org/apache/hadoop/hive/ql/reexec/ReOptimizePlugin.java > > Line 96 (original), 97 (patched) > > <https://reviews.apache.org/r/66663/diff/1-2/?file=2004568#file2004568line97> > > > > Cache size is used as number of entries. I see your intention but I am > > not sure it will work as intended. Also, its easier to reason about cache > > size as number of entries. So, lets use that.
okay..I've renamed the conf variable > On April 20, 2018, 1:20 a.m., Ashutosh Chauhan wrote: > > standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java > > Line 11643 (original), 11627 (patched) > > <https://reviews.apache.org/r/66663/diff/1-2/?file=2004593#file2004593line11643> > > > > Deletion shouldn't be based on total weight. It may skew entries from > > one large entry. > > It could be based on min weight, that is delete all entries whose > > weight < minWeight. the entries are ordered by createTime desc so the most recent totalWeight is preserved I don't think removing "small" entries would be beneficial...I think that small entries might be just as well usefull as large ones... > On April 20, 2018, 1:20 a.m., Ashutosh Chauhan wrote: > > standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/RawStore.java > > Line 1636 (original), 1636 (patched) > > <https://reviews.apache.org/r/66663/diff/1-2/?file=2004594#file2004594line1636> > > > > This method can be renamed : deleteRuntimeStats() ? sure > On April 20, 2018, 1:20 a.m., Ashutosh Chauhan wrote: > > standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java > > Lines 584 (patched) > > <https://reviews.apache.org/r/66663/diff/2/?file=2006697#file2006697line584> > > > > this wil turn off cleaner thread. Lets use default value of 86400 this is the max age of a runtime stat entry; I've disabled this to limit it only by number of runtime stats I've choosen to keep only the count limit; because I think there would be just a small number of queries which will be reoptimized in a real world scenario; and in those cases even a 1 month old stat would be probably accurate enough to create a plan which could be executed successfully - but this could have a non -1 default...should I set it to something like: 1-2 weeks? > On April 20, 2018, 1:20 a.m., Ashutosh Chauhan wrote: > > standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java > > Lines 586 (patched) > > <https://reviews.apache.org/r/66663/diff/2/?file=2006697#file2006697line586> > > > > sum of weights is little hard to reason about. Lets just have max > > weight as config. I've changed it to entries...it looks much clearer this way :) and really..weight is just a small detail from the implementation ...it would just confuse people trying to set this property - Zoltan ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/66663/#review201594 ----------------------------------------------------------- On April 19, 2018, 3:54 p.m., Zoltan Haindrich wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/66663/ > ----------------------------------------------------------- > > (Updated April 19, 2018, 3:54 p.m.) > > > Review request for hive and Ashutosh Chauhan. > > > Bugs: HIVE-19171 > https://issues.apache.org/jira/browse/HIVE-19171 > > > Repository: hive-git > > > Description > ------- > > * wip patch > * also contains HIVE-19009 > > > Diffs > ----- > > common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 73492ff99c > > itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java > 801de7aca2 > itests/src/test/resources/testconfiguration.properties d26f0ccb17 > itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java > fad8c0ffe9 > metastore/scripts/upgrade/derby/056-HIVE-19171.derby.sql PRE-CREATION > ql/src/java/org/apache/hadoop/hive/ql/Context.java 9ca8b0007e > ql/src/java/org/apache/hadoop/hive/ql/Driver.java f761fffbf1 > ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java > e15c5b7b66 > ql/src/java/org/apache/hadoop/hive/ql/optimizer/signature/OpSignature.java > e87bbceb7a > > ql/src/java/org/apache/hadoop/hive/ql/optimizer/signature/OpTreeSignature.java > c3dc848a32 > > ql/src/java/org/apache/hadoop/hive/ql/optimizer/signature/RuntimeStatsMap.java > PRE-CREATION > > ql/src/java/org/apache/hadoop/hive/ql/optimizer/signature/RuntimeStatsPersister.java > PRE-CREATION > ql/src/java/org/apache/hadoop/hive/ql/plan/FileSinkDesc.java e15a49f838 > ql/src/java/org/apache/hadoop/hive/ql/plan/HashTableSinkDesc.java > a61a47e390 > ql/src/java/org/apache/hadoop/hive/ql/plan/JoinDesc.java 5b7f4c3560 > ql/src/java/org/apache/hadoop/hive/ql/plan/MapJoinDesc.java 54b705db6e > ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/EmptyStatsSource.java > 72092ce4ae > ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/PlanMapper.java > a37280407d > > ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/SimpleRuntimeStatsSource.java > b5a3c2459f > ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/StatsSource.java > df5aa0c679 > ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/StatsSources.java > PRE-CREATION > ql/src/java/org/apache/hadoop/hive/ql/reexec/IReExecutionPlugin.java > 2b0d23c6f2 > ql/src/java/org/apache/hadoop/hive/ql/reexec/ReExecDriver.java 8a5595d210 > ql/src/java/org/apache/hadoop/hive/ql/reexec/ReExecutionOverlayPlugin.java > 4ee3c14b39 > ql/src/java/org/apache/hadoop/hive/ql/reexec/ReOptimizePlugin.java > f731315956 > ql/src/java/org/apache/hadoop/hive/ql/stats/OperatorStats.java 52e18a8030 > > ql/src/test/org/apache/hadoop/hive/ql/optimizer/signature/TestOperatorSignature.java > 0afc533da8 > > ql/src/test/org/apache/hadoop/hive/ql/optimizer/signature/TestRuntimeStatsPersistence.java > PRE-CREATION > ql/src/test/org/apache/hadoop/hive/ql/plan/mapping/TestCounterMapping.java > 18aeb3338b > ql/src/test/org/apache/hadoop/hive/ql/plan/mapping/TestReOptimization.java > b7263005ed > ql/src/test/queries/clientpositive/runtime_stats_hs2.q PRE-CREATION > ql/src/test/results/clientpositive/llap/runtime_stats_hs2.q.out > PRE-CREATION > standalone-metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore.h > 802d8e3fb2 > standalone-metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore.cpp > dfa13a0614 > > standalone-metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore_server.skeleton.cpp > c0a39f80e0 > standalone-metastore/src/gen/thrift/gen-cpp/hive_metastore_types.h > 2c95007daa > standalone-metastore/src/gen/thrift/gen-cpp/hive_metastore_types.cpp > 99024279c5 > > standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/RuntimeStat.java > PRE-CREATION > > standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ThriftHiveMetastore.java > a354f27cad > > standalone-metastore/src/gen/thrift/gen-php/metastore/ThriftHiveMetastore.php > 9c949429c5 > standalone-metastore/src/gen/thrift/gen-php/metastore/Types.php c4969d567f > > standalone-metastore/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore-remote > 079c7fc322 > > standalone-metastore/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore.py > d241414bc3 > standalone-metastore/src/gen/thrift/gen-py/hive_metastore/ttypes.py > 9bf9843314 > standalone-metastore/src/gen/thrift/gen-rb/hive_metastore_types.rb > 3dbe4d8068 > standalone-metastore/src/gen/thrift/gen-rb/thrift_hive_metastore.rb > 58ebd29523 > > standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java > ae9ec5cad8 > > standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java > 7ba053d3e8 > > standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java > 27f8775a10 > > standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java > a7acdcbc23 > > standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/RawStore.java > f6c46ee7bd > > standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/RuntimeStatsCleanerTask.java > PRE-CREATION > > standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java > ebdcbc237e > > standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java > 88b07cbdbf > > standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/model/MRuntimeStat.java > PRE-CREATION > standalone-metastore/src/main/resources/package.jdo 9ddf598d36 > standalone-metastore/src/main/sql/derby/hive-schema-3.0.0.derby.sql > 240a282f17 > standalone-metastore/src/main/sql/derby/upgrade-2.3.0-to-3.0.0.derby.sql > 334ddfadd9 > standalone-metastore/src/main/sql/mssql/hive-schema-3.0.0.mssql.sql > e15fc2e31c > standalone-metastore/src/main/sql/mssql/upgrade-2.3.0-to-3.0.0.mssql.sql > 53bdcc4f59 > standalone-metastore/src/main/sql/mysql/hive-schema-3.0.0.mysql.sql > f9efd56833 > standalone-metastore/src/main/sql/mysql/upgrade-2.3.0-to-3.0.0.mysql.sql > 1c540379d7 > standalone-metastore/src/main/sql/oracle/hive-schema-3.0.0.oracle.sql > a87e4464d5 > standalone-metastore/src/main/sql/oracle/upgrade-2.3.0-to-3.0.0.oracle.sql > 13b0d96bb4 > standalone-metastore/src/main/sql/postgres/hive-schema-3.0.0.postgres.sql > 07dd83d747 > > standalone-metastore/src/main/sql/postgres/upgrade-2.3.0-to-3.0.0.postgres.sql > ddb5e5009b > standalone-metastore/src/main/thrift/hive_metastore.thrift 5bba3294fa > > standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java > 304f567533 > > standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java > 85c67270d1 > > standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClientPreCatalog.java > 74c057bc26 > > standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/client/TestRuntimeStats.java > PRE-CREATION > > > Diff: https://reviews.apache.org/r/66663/diff/2/ > > > Testing > ------- > > > Thanks, > > Zoltan Haindrich > >
