----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/37963/ -----------------------------------------------------------
Review request for hive. Repository: hive-git Description ------- This is to fix the performance issues of graph walking during compilation. The PreOrderWalker walks the tree in DFS manner and for the cases of Lateral View in which one node could have multiple parents or children, it will repeat walking the child nodes which already visited. If there are multiple lateral views, the performance for a simple explain command is very bad. Implement a new LevelOrderWalker which only visit one node once. It can take necessary #levels of ancester nodes in the stack and only walk specified types of nodes to speed up performance. Diffs ----- ql/src/java/org/apache/hadoop/hive/ql/lib/DefaultGraphWalker.java 07d2734650f7a82f98dd2ddc03dcab6b5f01cac8 ql/src/java/org/apache/hadoop/hive/ql/lib/ForwardWalker.java 67b47004c06da82368cb6fdaa3f3e1210646f57c ql/src/java/org/apache/hadoop/hive/ql/lib/LevelOrderWalker.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/lib/PreOrderWalker.java f22694b6d8f6fb0e4d29694813a4c3b19db04cc3 ql/src/java/org/apache/hadoop/hive/ql/optimizer/ColumnPruner.java 735b448a97ce3b83c018dbf3285851242734aeb9 ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConstantPropagate.java b6f1f272712f3998c27709f2b132be28e401846c ql/src/java/org/apache/hadoop/hive/ql/optimizer/lineage/Generator.java 9a5cf55609708d04f36d81bdcee952c623e4108c ql/src/java/org/apache/hadoop/hive/ql/optimizer/metainfo/annotation/AnnotateWithOpTraits.java c304e97aebfb3a9a4b456f6a8f591a7a3bee60ae ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/AnnotateWithStatistics.java 4aeeff20d6c13531f55f60e5252fd8f57ae0efdd ql/src/java/org/apache/hadoop/hive/ql/optimizer/unionproc/UnionProcessor.java 9937343184d6d0b6f3ec9b992ced212fa43dafd7 ql/src/java/org/apache/hadoop/hive/ql/parse/GenMapRedWalker.java 9583a1b82cdf01da456b5948ab4f42627d72a15b ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezWorkWalker.java 2d8c8b20c3393f41a235d9f091d15f3313cb4e06 ql/src/java/org/apache/hadoop/hive/ql/parse/TezWalker.java 2f63c1aabc3422b8f3c472634291b5da49b7e239 ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkWorkWalker.java e31c025a7e6d4f4db53f1936e70f2c49f82e7020 ql/src/java/org/apache/hadoop/hive/ql/ppd/PredicateTransitivePropagate.java ea1f7136292b8a965ffc640e3d78827f4e1e4c59 ql/src/test/results/clientpositive/correlationoptimizer5.q.out 63741fc6eb18d10fe635e4b7b978ce602d34c855 ql/src/test/results/clientpositive/join32.q.out 579566907ea03d4fc61c90bfc711c9cccd04c534 ql/src/test/results/clientpositive/join32_lessSize.q.out c027dbaf5c222d128d55a366f61356a43bef9837 ql/src/test/results/clientpositive/join33.q.out 579566907ea03d4fc61c90bfc711c9cccd04c534 ql/src/test/results/clientpositive/lineage2.q.out 9b227c649df085714ad6c33fbdddcb9d02be61c0 ql/src/test/results/clientpositive/lineage3.q.out b6b4e0bc46d1819cebf356e8074a35103369b61c ql/src/test/results/clientpositive/spark/join32.q.out 4ae9dc6e3d5673f92908457cecb25bf0cc153ea6 ql/src/test/results/clientpositive/spark/join32_lessSize.q.out 78bb6554d609a0b5949524b7d240f155ac86c02e ql/src/test/results/clientpositive/spark/join33.q.out 4ae9dc6e3d5673f92908457cecb25bf0cc153ea6 ql/src/test/results/clientpositive/spark/subquery_multiinsert.q.java1.7.out 1bfdba20995aa48dc2981cefe553170a93431d67 ql/src/test/results/clientpositive/subquery_multiinsert.q.java1.7.out 3a2473fd37ed9e919a63eb281148215d9203e9b8 ql/src/test/results/clientpositive/tez/explainuser_2.q.out 71d8f417be2dc4dafe75b2bc2283637b84493cf3 Diff: https://reviews.apache.org/r/37963/diff/ Testing ------- Thanks, Aihua Xu
