> On Nov. 5, 2014, 9:23 p.m., Szehon Ho wrote: > > ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/SparkMapJoinResolver.java, > > line 201 > > <https://reviews.apache.org/r/27640/diff/1/?file=750693#file750693line201> > > > > We need to handle the case of where parent is the mapwork of big-table. > > Basically if we are walking from big-table to a reduce work of mapjoin, we > > should not replace RS with HTS. (it can happen for example, if big-table > > is the result from a group by). > > > > Only work of small-tables should get replaced.
Good point, I will add a check for this. > On Nov. 5, 2014, 9:23 p.m., Szehon Ho wrote: > > ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/SparkMapJoinResolver.java, > > line 202 > > <https://reviews.apache.org/r/27640/diff/1/?file=750693#file750693line202> > > > > Let's just get mapJoinOp and check if its null in one call. (Can get > > rid of one of the methods). agreed, will fix this. > On Nov. 5, 2014, 9:23 p.m., Szehon Ho wrote: > > ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/SparkMapJoinResolver.java, > > line 244 > > <https://reviews.apache.org/r/27640/diff/1/?file=750693#file750693line244> > > > > Please fix all these tabs, and make sure to indent them properly. Sorry about that, will do. Need to change my IDE default settings > On Nov. 5, 2014, 9:23 p.m., Szehon Ho wrote: > > ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/SparkMapJoinResolver.java, > > line 254 > > <https://reviews.apache.org/r/27640/diff/1/?file=750693#file750693line254> > > > > Are you sure we dont need to initialize the HTSOperator's values like > > it does in LocalMapJoinProcFactory? I will take a closer look. - Suhas ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/27640/#review60031 ----------------------------------------------------------- On Nov. 5, 2014, 8:29 p.m., Suhas Satish wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/27640/ > ----------------------------------------------------------- > > (Updated Nov. 5, 2014, 8:29 p.m.) > > > Review request for hive, Chao Sun, Jimmy Xiang, Szehon Ho, and Xuefu Zhang. > > > Repository: hive-git > > > Description > ------- > > This replaces ReduceSinks with HashTableSinks in smaller tables for a > map-join. But the condition check field to detect map-join is actually being > set in CommonJoinResolver, which doesnt exist yet. We need to decide where is > the right place to populate this field. > > > Diffs > ----- > > > ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/SparkMapJoinResolver.java > PRE-CREATION > ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkCompiler.java > 795a5d7 > > Diff: https://reviews.apache.org/r/27640/diff/ > > > Testing > ------- > > > Thanks, > > Suhas Satish > >
