> On July 12, 2018, 5:18 p.m., Ashutosh Chauhan wrote: > > ql/src/java/org/apache/hadoop/hive/ql/ppd/SyntheticJoinPredicate.java > > Lines 296 (patched) > > <https://reviews.apache.org/r/67887/diff/2/?file=2058961#file2058961line302> > > > > Can we always look past Gby? Shall we restrict it to map side GBy? What > > about rollup and grouping sets which are also represented by Gby op?
I think so. The idea is that the GBy will not generate any new values, except NULL for the grouping sets variant. Those NULL values will not join upstream upstream in the plan, hence whatever was coming from below the GBy are the only possible values that will be joined upstream. Does it make sense? > On July 12, 2018, 5:18 p.m., Ashutosh Chauhan wrote: > > ql/src/java/org/apache/hadoop/hive/ql/ppd/SyntheticJoinPredicate.java > > Lines 312 (patched) > > <https://reviews.apache.org/r/67887/diff/2/?file=2058961#file2058961line318> > > > > Leave a TODO since other simple expressions like CAST can also be > > supported I have added the TODO, though I think if we pass through CBO we will never have expressions over here that are more complex than a column (except for maybe a constant) because they will be computed below with a SelectOp because of Calcite generated plan. > On July 12, 2018, 5:18 p.m., Ashutosh Chauhan wrote: > > ql/src/test/results/clientpositive/perf/tez/query32.q.out > > Lines 62-64 (original), 62 (patched) > > <https://reviews.apache.org/r/67887/diff/2/?file=2058976#file2058976line62> > > > > Why are these vertices gone? Extra packing of operators in vertices by > > SharedWork opt? We create a new semijoin filter : _(cs_item_sk BETWEEN DynamicValue(RS_24_item_i_item_sk_min) AND DynamicValue(RS_24_item_i_item_sk_max) and in_bloom_filter(cs_item_sk, DynamicValue(RS_24_item_i_item_sk_bloom_filter)))_ in FIL-121 in the new plan. This was not present before on one of the branches, and it leads to reutilization of the complete subtree up to MERGEJOIN-101. > On July 12, 2018, 5:18 p.m., Ashutosh Chauhan wrote: > > ql/src/test/results/clientpositive/perf/tez/query64.q.out > > Lines 243 (patched) > > <https://reviews.apache.org/r/67887/diff/2/?file=2058981#file2058981line243> > > > > Does this mean we Sharedwork opt didnt pack this Map with other map? This query is so difficult... There is certainly a new semijoin edge coming from item to catalog_sales (in particular, this is the reason why we introduced the optimization). In turn, this is preventing some reutilization opportunities, probably because we are creating additional cycles due to this edge. > On July 12, 2018, 5:18 p.m., Ashutosh Chauhan wrote: > > ql/src/test/results/clientpositive/perf/tez/query92.q.out > > Line 74 (original), 71 (patched) > > <https://reviews.apache.org/r/67887/diff/2/?file=2058988#file2058988line74> > > > > I couldn't understand these plan changes. Can you explain whats going > > on? > > I see that there is an extra semijoin edge going into Map 1 from R7 > > which is expected because of patch. However, why 3 vertices are now > > disappearing? Is it because of shared work opt? I think it is a combination of multiple things. SharedWorkOptimizer finds new reutilization opportunities: web_sales joined with date_dim is reused. - Jesús ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/67887/#review206015 ----------------------------------------------------------- On July 12, 2018, 3:55 p.m., Jesús Camacho Rodríguez wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/67887/ > ----------------------------------------------------------- > > (Updated July 12, 2018, 3:55 p.m.) > > > Review request for hive, Ashutosh Chauhan, Deepak Jaiswal, and Gopal V. > > > Bugs: HIVE-20090 > https://issues.apache.org/jira/browse/HIVE-20090 > > > Repository: hive-git > > > Description > ------- > > HIVE-20090 > > > Diffs > ----- > > common/src/java/org/apache/hadoop/hive/conf/HiveConf.java > 6ea68c35000a5dadb7a01db47bbd8183bff966da > itests/src/test/resources/testconfiguration.properties > 4001b9f452f9dbeaff31c2e766334259605a51af > ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java > 119aa925c1a71502e649b4f2d193a7ff974263c1 > ql/src/java/org/apache/hadoop/hive/ql/ppd/SyntheticJoinPredicate.java > dec2d1ef38b748a5c9b40d06af491dd168d70b72 > ql/src/test/queries/clientpositive/dynamic_semijoin_reduction_sw2.q > PRE-CREATION > > ql/src/test/results/clientpositive/llap/dynamic_semijoin_reduction_sw2.q.out > PRE-CREATION > ql/src/test/results/clientpositive/llap/explainuser_1.q.out > f87fe36e11a7c7e535678dbfaaced04f33bbb501 > ql/src/test/results/clientpositive/llap/tez_fixed_bucket_pruning.q.out > 6987a96809e3c3300e1b76ea5df3069b3c1d162f > ql/src/test/results/clientpositive/perf/tez/query1.q.out > 579940c66e25ebf5e7d0635aaedd0c0cc994f4e0 > ql/src/test/results/clientpositive/perf/tez/query16.q.out > 0b64c55b0f4ba036aeba4c49f478e9ee1409087c > ql/src/test/results/clientpositive/perf/tez/query17.q.out > 2e5e254b2ddc3507f962cbc7691db51f1abafbca > ql/src/test/results/clientpositive/perf/tez/query18.q.out > e8585275b4e51a55ce778dd154033fcdf859e617 > ql/src/test/results/clientpositive/perf/tez/query2.q.out > d24899ccf371ad42ef88cebc26cc671c097686da > ql/src/test/results/clientpositive/perf/tez/query23.q.out > 6725bec30106bc3321c2869dfc304d0a4da82cf8 > ql/src/test/results/clientpositive/perf/tez/query24.q.out > 9fcec42c3ab29b898c9c947544a2e29dd08e95e8 > ql/src/test/results/clientpositive/perf/tez/query25.q.out > a885cf344b7e29dcf1b2d93d1914e7f9a8d4b921 > ql/src/test/results/clientpositive/perf/tez/query29.q.out > 46ff49d41a01591f075b2c48ae5a692640fd6eec > ql/src/test/results/clientpositive/perf/tez/query31.q.out > c4d717d8680f6ac6f8f8b6ed01742384a84ddcf9 > ql/src/test/results/clientpositive/perf/tez/query32.q.out > 6be6f7aa6e6fc50bcedebe3f4d1b5fc00b52ee86 > ql/src/test/results/clientpositive/perf/tez/query39.q.out > 5966e243ea79b4b884950f34a5b7336e40f92889 > ql/src/test/results/clientpositive/perf/tez/query40.q.out > 2f116f12ebcba44b876508d0d0f0d827e3a8b28d > ql/src/test/results/clientpositive/perf/tez/query54.q.out > 8ab239ce260fb37d988d956fcb9e4eb98a3aeb88 > ql/src/test/results/clientpositive/perf/tez/query59.q.out > 6b2dcc38737cfc9b955cca1d5b1ac99a7901370b > ql/src/test/results/clientpositive/perf/tez/query64.q.out > a673b9f753a641e111e30a7a4427206d5f2c3da3 > ql/src/test/results/clientpositive/perf/tez/query69.q.out > a9c7ac3b21b3e0588e7df7e8c2129fc641d090f1 > ql/src/test/results/clientpositive/perf/tez/query72.q.out > 48682e340db2916800e9bc5ad61c08c0fb4a8a8b > ql/src/test/results/clientpositive/perf/tez/query77.q.out > 163805b2a3dba3e4169d487bd44e7906f66e5868 > ql/src/test/results/clientpositive/perf/tez/query78.q.out > 90b6f17e1d10ca1e3af17bc53b6df50ffa310af4 > ql/src/test/results/clientpositive/perf/tez/query80.q.out > 816b525c301fe74460e5657d0b230287d0a6729f > ql/src/test/results/clientpositive/perf/tez/query91.q.out > 5e0f00a3e7321c4233f927703701051cab641fb0 > ql/src/test/results/clientpositive/perf/tez/query92.q.out > 061fcf729d6fa7fde52de3ccd46a800379a92211 > ql/src/test/results/clientpositive/perf/tez/query94.q.out > 5d19a1634b4657e9ef9595891401e8831d9b0bd4 > ql/src/test/results/clientpositive/perf/tez/query95.q.out > 400cc1958116b2347a06b52a1460320fd0e0be43 > > ql/src/test/results/clientpositive/spark/spark_dynamic_partition_pruning_3.q.out > eafc1c4a005fa2b3bc169aa4453376f5da6841bc > > > Diff: https://reviews.apache.org/r/67887/diff/2/ > > > Testing > ------- > > > Thanks, > > Jesús Camacho Rodríguez > >
