> On Nov. 11, 2018, 3:25 a.m., Ashutosh Chauhan wrote: > > ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java > > Lines 1531 (patched) > > <https://reviews.apache.org/r/69266/diff/1/?file=2104334#file2104334line1535> > > > > Should this logic be run as part of StatsRulesProcFactory when its > > computing stats for TS, since DPP branches are already created then. > > This will ensure that op tree's stats are updated for DPPso all walkers > > on tree will see this. Also all downstream ops will also compute there > > stats with DPP for TS being accounted for in StatsRulesProcFactory in that > > case. As its currently written these stats are visible only after SJ rules.
I have created HIVE-20957 as a follow-up. > On Nov. 11, 2018, 3:25 a.m., Ashutosh Chauhan wrote: > > ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java > > Lines 1541 (patched) > > <https://reviews.apache.org/r/69266/diff/1/?file=2104334#file2104334line1545> > > > > Can't there be a SEL here and FIL following that? I have removed this logic in the final patch, as it is part of the DPP. In any case, this should be a Filter operator if there is a SJ targetting that TS too, since the SJ Filter will always be just on top of the TS. > On Nov. 11, 2018, 3:25 a.m., Ashutosh Chauhan wrote: > > ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java > > Lines 1567 (patched) > > <https://reviews.apache.org/r/69266/diff/1/?file=2104334#file2104334line1571> > > > > This is updating stats for FIL op. But won't we need to retrigger > > updates on all downstream ops? Same as above, we should take care of this in HIVE-20957. > On Nov. 11, 2018, 3:25 a.m., Ashutosh Chauhan wrote: > > ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java > > Line 1560 (original), 1615 (patched) > > <https://reviews.apache.org/r/69266/diff/1/?file=2104334#file2104334line1619> > > > > Does this need instanceof check? Since this is a TS reduced by a SJ, we know that we will always have a Filter operator on top of the TS (observe that the cast was already there in the original code, just as a parameter for the computeBloomFilterNetBenefit method. > On Nov. 11, 2018, 3:25 a.m., Ashutosh Chauhan wrote: > > ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java > > Lines 1628-1645 (patched) > > <https://reviews.apache.org/r/69266/diff/1/?file=2104334#file2104334line1632> > > > > I don't follow this logic and role of reductionFactorMap. Can you > > please add comments for it? Done. > On Nov. 11, 2018, 3:25 a.m., Ashutosh Chauhan wrote: > > ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java > > Lines 1662 (patched) > > <https://reviews.apache.org/r/69266/diff/1/?file=2104334#file2104334line1666> > > > > Do we need to trigger updateStats() for downstream ops? Same as above, we should take care of this in HIVE-20957, currently we do not propagate the update. - Jesús ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/69266/#review210454 ----------------------------------------------------------- On Nov. 7, 2018, 12:18 a.m., Jesús Camacho Rodríguez wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/69266/ > ----------------------------------------------------------- > > (Updated Nov. 7, 2018, 12:18 a.m.) > > > Review request for hive, Ashutosh Chauhan and Deepak Jaiswal. > > > Bugs: HIVE-20775 > https://issues.apache.org/jira/browse/HIVE-20775 > > > Repository: hive-git > > > Description > ------- > > HIVE-20783 > > > Diffs > ----- > > > ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java > 32fba6c8ff80befdde55542a4ae83b619256632e > ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java > 91d2f1f09112b1fc73dc0f9d4ed2784880f7a721 > ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java > b7adc485a70e148e71feb594f311bfad1763479d > ql/src/test/results/clientpositive/llap/dynamic_semijoin_reduction.q.out > ff9d98c63efb894d0503ec16d0ab1e8005fa8f7e > ql/src/test/results/clientpositive/llap/dynamic_semijoin_reduction_4.q.out > cc1c06da6346950155cd37dba5b5711c2e582b2e > ql/src/test/results/clientpositive/llap/dynamic_semijoin_reduction_sw.q.out > bd7fcbd7951423094cfd8e960645773da2dba903 > > ql/src/test/results/clientpositive/llap/dynamic_semijoin_reduction_sw2.q.out > abcbd9727a9502a2007ae91a59fa0c44e063b4e8 > ql/src/test/results/clientpositive/llap/dynamic_semijoin_user_level.q.out > bc9e6fb083e73cb9c2532c79c0db3997790e6bf4 > > ql/src/test/results/clientpositive/llap/vectorized_dynamic_semijoin_reduction.q.out > 89986fbb3065aa87e4504711c99cd796f3bd1f8d > ql/src/test/results/clientpositive/perf/tez/cbo_query23.q.out > ace7cf5b791fe6ff98d9d5055dc9022674225655 > ql/src/test/results/clientpositive/perf/tez/cbo_query54.q.out > eaf25363b166bc2105f64791a857707465ff2251 > ql/src/test/results/clientpositive/perf/tez/constraints/cbo_query54.q.out > 1cf3ce40745102346aa1f3496310be0cbcd7d4e3 > ql/src/test/results/clientpositive/perf/tez/constraints/query10.q.out > 3fbd92878e0197ec0db1ce808f9bc4c0f5b255a3 > ql/src/test/results/clientpositive/perf/tez/constraints/query12.q.out > 741bd90666c033a6874d1b2299a9404adf7e0ba4 > ql/src/test/results/clientpositive/perf/tez/constraints/query13.q.out > 02966e4f474c8247e85230d24a3aee2b18962bd9 > ql/src/test/results/clientpositive/perf/tez/constraints/query14.q.out > e8a6eaa464c17e2adeae3cb03ea0a8b083c1cef7 > ql/src/test/results/clientpositive/perf/tez/constraints/query16.q.out > 3143be8480647bbbf47f13c12f83248980df4b95 > ql/src/test/results/clientpositive/perf/tez/constraints/query17.q.out > e796101e4527f3ad418e28b7f93b3134ad4f8fc7 > ql/src/test/results/clientpositive/perf/tez/constraints/query18.q.out > b7f97780c7d9472f2f237860e4f29eb1b92592c3 > ql/src/test/results/clientpositive/perf/tez/constraints/query19.q.out > 7eb52efbf48309f545e5bfbd305434b7ce1de3f5 > ql/src/test/results/clientpositive/perf/tez/constraints/query20.q.out > da3e262352cab9fc5294af87e06cf7f1b0376fec > ql/src/test/results/clientpositive/perf/tez/constraints/query23.q.out > 292c920583f354cdb6ad48dd74229d4799299ae7 > ql/src/test/results/clientpositive/perf/tez/constraints/query24.q.out > fb77386d6ef9f72b4fff4148b824b37779383398 > ql/src/test/results/clientpositive/perf/tez/constraints/query25.q.out > a353c6a12869bdb720fdf27b1d0f55064e218950 > ql/src/test/results/clientpositive/perf/tez/constraints/query26.q.out > ed032b1b66b12fb5a83df8c0475c500c38eee940 > ql/src/test/results/clientpositive/perf/tez/constraints/query27.q.out > fdc1791a13450cd3852ac7935da569c8add4f11e > ql/src/test/results/clientpositive/perf/tez/constraints/query29.q.out > 715f1cef532507340d66fa27a5d01b6aff73aea1 > ql/src/test/results/clientpositive/perf/tez/constraints/query3.q.out > 0cca24c23c34718b3ff5ccf9f7b179376633c86f > ql/src/test/results/clientpositive/perf/tez/constraints/query32.q.out > d909c0254d5bcd239517c9647026f1515673a049 > ql/src/test/results/clientpositive/perf/tez/constraints/query33.q.out > 6d7c620deac47a87fe8707696988855c7cc8319c > ql/src/test/results/clientpositive/perf/tez/constraints/query34.q.out > f4e6a73942b0a7ab424cbb5b4e2b529c10150e40 > ql/src/test/results/clientpositive/perf/tez/constraints/query35.q.out > 2501199e8955cecbc36f28a5eb70c562939610f0 > ql/src/test/results/clientpositive/perf/tez/constraints/query36.q.out > f2c0b4b0eecf7ab04b2c7b505d697e50306d27a0 > ql/src/test/results/clientpositive/perf/tez/constraints/query37.q.out > fb4cbf4e985e6c2e772b2c5e741fc5cbe0e1f8dc > ql/src/test/results/clientpositive/perf/tez/constraints/query40.q.out > c05256fbd9f32e82ebe3972a70d70fd2d8858b17 > ql/src/test/results/clientpositive/perf/tez/constraints/query42.q.out > 02b9f7c07453f68dba4c1ce3fdf70f74cd791ec1 > ql/src/test/results/clientpositive/perf/tez/constraints/query43.q.out > 495b6bd6261b34060b431871c986c03f90e3652b > ql/src/test/results/clientpositive/perf/tez/constraints/query46.q.out > b7a6bd626c536b51fab97a4856cc8555b7e3ee4f > ql/src/test/results/clientpositive/perf/tez/constraints/query48.q.out > b84dfce073b982a693933e4624e5aa44b10ff044 > ql/src/test/results/clientpositive/perf/tez/constraints/query50.q.out > 05d84d862103a416e58a8916dc075cfed2dfcfa1 > ql/src/test/results/clientpositive/perf/tez/constraints/query52.q.out > 72f91517ecafc6d80bb0c8b4280a75694532588a > ql/src/test/results/clientpositive/perf/tez/constraints/query53.q.out > 27adc6ec66f00e5f446603bf6f69fb7288b6cb76 > ql/src/test/results/clientpositive/perf/tez/constraints/query54.q.out > 0210163c8adf5f680134aba07b0768deedb11351 > ql/src/test/results/clientpositive/perf/tez/constraints/query55.q.out > eb6b84fd0972f14cb4c97524ce88c68f3047e5d3 > ql/src/test/results/clientpositive/perf/tez/constraints/query56.q.out > cac7668b884758195ae737879856384ee34d68bf > ql/src/test/results/clientpositive/perf/tez/constraints/query60.q.out > 5ba912a10557fe155a582524cdd9d1512ae9dc4e > ql/src/test/results/clientpositive/perf/tez/constraints/query61.q.out > 95a2a747cde36b10a50feff46652b96aa5051e29 > ql/src/test/results/clientpositive/perf/tez/constraints/query63.q.out > a1f8413eb02fe5fee781259aef9933843dcdd9c5 > ql/src/test/results/clientpositive/perf/tez/constraints/query64.q.out > aebd6b4137d3106dde757d41c0e449e73f8b3c11 > ql/src/test/results/clientpositive/perf/tez/constraints/query65.q.out > 87d0f6fc8777361c2678c92c841305e4072eb22e > ql/src/test/results/clientpositive/perf/tez/constraints/query66.q.out > f82272c3f2a38430c1a9db4e30fda4acaa21c33f > ql/src/test/results/clientpositive/perf/tez/constraints/query68.q.out > 2188af561e9a772fea94663fda77a98f57a5b07d > ql/src/test/results/clientpositive/perf/tez/constraints/query69.q.out > 8fe313495fa2ea2ce850b86959822df28ba94181 > ql/src/test/results/clientpositive/perf/tez/constraints/query7.q.out > 6cfaac639f07edfb9634f27958fb71ebf018e8f5 > ql/src/test/results/clientpositive/perf/tez/constraints/query71.q.out > 81f7961df9254d078ca60246bedaac0d264a651c > ql/src/test/results/clientpositive/perf/tez/constraints/query72.q.out > f27fbc9273303cdb30a7d86fb2c9508a736cd033 > ql/src/test/results/clientpositive/perf/tez/constraints/query73.q.out > 45ddfd5a4a0ec6fd2a0c1d31300aa2d6d2cf5cff > ql/src/test/results/clientpositive/perf/tez/constraints/query75.q.out > f2593995b85be460feb5bc9de92e97417b21ead3 > ql/src/test/results/clientpositive/perf/tez/constraints/query79.q.out > fb94428c0345b85a4a93023dbd1df6d8a65c394e > ql/src/test/results/clientpositive/perf/tez/constraints/query8.q.out > 93bce2e0599413f85c20c0904efc3ace194bca6c > ql/src/test/results/clientpositive/perf/tez/constraints/query80.q.out > dbaecf8fe8f08d61feec87ff3d1989d71c477679 > ql/src/test/results/clientpositive/perf/tez/constraints/query82.q.out > 1344b9422a5ec4a69a6dd772cf114e70661cc87c > ql/src/test/results/clientpositive/perf/tez/constraints/query88.q.out > 08079cd2440a32e79c9b6e4a7ed744d057a110df > ql/src/test/results/clientpositive/perf/tez/constraints/query89.q.out > 673050eee7405c526c241c6dcb729612b75eb0e3 > ql/src/test/results/clientpositive/perf/tez/constraints/query90.q.out > f902607983614eca0bd62fdf7141aad0227dd590 > ql/src/test/results/clientpositive/perf/tez/constraints/query92.q.out > da5a94e126826860bdb9632efa02793df3d1fa85 > ql/src/test/results/clientpositive/perf/tez/constraints/query93.q.out > 2d9ea2eb76578409ca5fb72700340387a9023d12 > ql/src/test/results/clientpositive/perf/tez/constraints/query94.q.out > 8c813d84eea8f1d0e6dafc8badaed039a537c88e > ql/src/test/results/clientpositive/perf/tez/constraints/query95.q.out > fd709f99dd19f19560081ae1858690d948aafc49 > ql/src/test/results/clientpositive/perf/tez/constraints/query96.q.out > 8942a90185a00ef7c826a0a36bbbe43d59821b1c > ql/src/test/results/clientpositive/perf/tez/constraints/query98.q.out > 3ffc379cd93b75ab9ea0856fa00db0c7d497ef8e > ql/src/test/results/clientpositive/perf/tez/constraints/query99.q.out > 9b8ebd20b261a4f111a175dc26c24868f465e562 > ql/src/test/results/clientpositive/perf/tez/query10.q.out > d049b2f28bae00a59ce0793dd5adbcf5676c0beb > ql/src/test/results/clientpositive/perf/tez/query12.q.out > 75156be9fdc52b7b72f797a3553623a22512a99d > ql/src/test/results/clientpositive/perf/tez/query13.q.out > 0e02e9aaa15e24ee2371b78f172298d06290b8a8 > ql/src/test/results/clientpositive/perf/tez/query14.q.out > c078c271ecc3c65ff00ebea906560aa92475946b > ql/src/test/results/clientpositive/perf/tez/query16.q.out > b09d953c4b2592420e22f292f63e2c36158b3f87 > ql/src/test/results/clientpositive/perf/tez/query17.q.out > fa576c3479d6582456d76b5578e34ab3fb4d04c2 > ql/src/test/results/clientpositive/perf/tez/query18.q.out > 58fb7a79f52690923194bfe6b6d5cc0ff342ce56 > ql/src/test/results/clientpositive/perf/tez/query19.q.out > fd8578f40262014d65b29fd896f98f65deaee662 > ql/src/test/results/clientpositive/perf/tez/query20.q.out > 892beb37cb3ca0a737777e0abe7e89dfc13804a3 > ql/src/test/results/clientpositive/perf/tez/query23.q.out > 77847929b4045ec93bac03af9e2c2e3fa39c52b4 > ql/src/test/results/clientpositive/perf/tez/query24.q.out > 43ece8527571c8901c71dd376a47e1dc042cabbd > ql/src/test/results/clientpositive/perf/tez/query25.q.out > 5d1c9fc1dfd9b9ac47fbcd2ba9231cd77a556a91 > ql/src/test/results/clientpositive/perf/tez/query26.q.out > 7c42069758b9e81cb1c639e32547a5ad4c396fbf > ql/src/test/results/clientpositive/perf/tez/query27.q.out > 59cca4f94f4a2a82c7e1853c0f33f80f39970914 > ql/src/test/results/clientpositive/perf/tez/query29.q.out > a21c3c789e382ebe2e3db5b408f9dcd3bf8e67fa > ql/src/test/results/clientpositive/perf/tez/query3.q.out > d4296cf3b23ec4b9d736f7702a0101cd2d415004 > ql/src/test/results/clientpositive/perf/tez/query32.q.out > 1c1a2e7b5e1509c259fd2065e880a1154eca6e35 > ql/src/test/results/clientpositive/perf/tez/query33.q.out > a76122c4edba49fe0ffce5fb14f54cd4d7cf4fe0 > ql/src/test/results/clientpositive/perf/tez/query34.q.out > fa40be9bb94be42f0d4096624e4e822027e07c08 > ql/src/test/results/clientpositive/perf/tez/query35.q.out > 44993831f41c0125815aa18e7bc0b9cd35f7b174 > ql/src/test/results/clientpositive/perf/tez/query36.q.out > 4f4ce8a518f8760b22dc862a74dc523466f3fc63 > ql/src/test/results/clientpositive/perf/tez/query37.q.out > 4407a489311039912ce4794e692e91344dee688b > ql/src/test/results/clientpositive/perf/tez/query40.q.out > 1cca18d3069731b3dbe08e5232ba7d9f68832dd0 > ql/src/test/results/clientpositive/perf/tez/query42.q.out > a458f5e095aa0ac21de46341ea2a615415790eca > ql/src/test/results/clientpositive/perf/tez/query43.q.out > f75929bbb754c730fdd82ae0e693ffe12da5f649 > ql/src/test/results/clientpositive/perf/tez/query46.q.out > 6d394e73175233f77702ced690c3d7dcc83d50ad > ql/src/test/results/clientpositive/perf/tez/query48.q.out > 76b4ce1fe181a7b29776c0493656c7d75325fda4 > ql/src/test/results/clientpositive/perf/tez/query50.q.out > ae6781ba58e93a3bc76177f5b84b2d170d35aa7c > ql/src/test/results/clientpositive/perf/tez/query52.q.out > bc932b2deca1250ab6281cd3899c443006fe1482 > ql/src/test/results/clientpositive/perf/tez/query53.q.out > d99529f1a55ea9ad6f5eac11b49efa6006d2ec1c > ql/src/test/results/clientpositive/perf/tez/query54.q.out > a32f264d8b7b5a9e07c8866f80cd5513e038ffb4 > ql/src/test/results/clientpositive/perf/tez/query55.q.out > ec30b92cc622027565adf1cfb94ab2babda5176d > ql/src/test/results/clientpositive/perf/tez/query56.q.out > a6d3090c08c36977d1236c81b311905de9ccd6cb > ql/src/test/results/clientpositive/perf/tez/query60.q.out > f94101a4c7fd6ca94dc3702e7eb1bfb419c195c2 > ql/src/test/results/clientpositive/perf/tez/query61.q.out > dc18d84d46d077d39bf58125b897312e69fcec35 > ql/src/test/results/clientpositive/perf/tez/query63.q.out > 6a6ffb7e3d2998843ce2751720822a5ca7ece02f > ql/src/test/results/clientpositive/perf/tez/query64.q.out > f670c4f4e21f59b90c8f6c88fa2306e86026261d > ql/src/test/results/clientpositive/perf/tez/query65.q.out > 4c3644b16a219bc4316c19d4714b1018388fe340 > ql/src/test/results/clientpositive/perf/tez/query66.q.out > 225b62f7e268605baed028d41a77818a94d71e3f > ql/src/test/results/clientpositive/perf/tez/query68.q.out > 7c94381d40f2dbc1db18cdd4f8d26650439c46b5 > ql/src/test/results/clientpositive/perf/tez/query69.q.out > 65601d745c44909f48406295271d158c94579a68 > ql/src/test/results/clientpositive/perf/tez/query7.q.out > 6cd8fb58f45fbed32d34939aef531593ca16b875 > ql/src/test/results/clientpositive/perf/tez/query71.q.out > e031c3aa054c5c768d9f8236ede2c8c6de8f2d5d > ql/src/test/results/clientpositive/perf/tez/query72.q.out > 700a8769a2986eff4636567384ccdfb4e05ad78f > ql/src/test/results/clientpositive/perf/tez/query73.q.out > 7aafbcf61c2ccb73a1d84faded5be9b12f0177cd > ql/src/test/results/clientpositive/perf/tez/query75.q.out > 9968adef414fa9a8017ee2e1b2be9917d72c298d > ql/src/test/results/clientpositive/perf/tez/query79.q.out > f1538371a9997c280c4c508fea0f639d2b155606 > ql/src/test/results/clientpositive/perf/tez/query8.q.out > 8c245509ca7711fa1ea48d4a1fdf9b15022fb928 > ql/src/test/results/clientpositive/perf/tez/query80.q.out > ff26befbd41d73a51a7a9f1432e282274ad8a4e0 > ql/src/test/results/clientpositive/perf/tez/query82.q.out > 6bc9666784fc90da3e77e99219510f30d679b548 > ql/src/test/results/clientpositive/perf/tez/query88.q.out > 28a970755c750b4e8b04a6004591075bdd1959ac > ql/src/test/results/clientpositive/perf/tez/query89.q.out > 0844ca79a8294d37eb40dabb467dc75e164aa51b > ql/src/test/results/clientpositive/perf/tez/query90.q.out > 653ce1a912b323ec26fcc0220553367a169ca060 > ql/src/test/results/clientpositive/perf/tez/query92.q.out > 50918f0966c097436a5e49e9969b00bf202ed022 > ql/src/test/results/clientpositive/perf/tez/query93.q.out > 2891fccb3f5afcdbe18e39a3791b42a9f1888da2 > ql/src/test/results/clientpositive/perf/tez/query94.q.out > ada9f3b86b7f7bb94802289dd78442dd4b99f7a9 > ql/src/test/results/clientpositive/perf/tez/query95.q.out > 49e8e868c68e8103a82a81a65fe3fb2261bbaff6 > ql/src/test/results/clientpositive/perf/tez/query96.q.out > d06ad6a611d1401a7e6618312391b4452204d2f5 > ql/src/test/results/clientpositive/perf/tez/query98.q.out > 8403d20e704a7e723d4d13a4d4ce78c4f58d8897 > ql/src/test/results/clientpositive/perf/tez/query99.q.out > 745c5a9c4b02a057c011704b5aa98b1f9bfc83a9 > > > Diff: https://reviews.apache.org/r/69266/diff/1/ > > > Testing > ------- > > > Thanks, > > Jesús Camacho Rodríguez > >
