> On July 22, 2013, 10:05 p.m., Ashutosh Chauhan wrote: > > ql/src/java/org/apache/hadoop/hive/ql/optimizer/ppr/PartitionPruner.java, > > line 353 > > <https://reviews.apache.org/r/12705/diff/1/?file=322919#file322919line353> > > > > Why are we restricting this for strict mode? We should skip default > > partition in all cases unless explicitly requested by user. Assumption is > > default partition contains rows which were malformed in some ways at load > > times and will be excluded from all further query processing.
I am being defensive here as the change is drastic. With the current code the behavior change only affects strict mode. Let me know if i should do this generically? - John ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/12705/#review23657 ----------------------------------------------------------- On July 17, 2013, 10:19 p.m., John Pullokkaran wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/12705/ > ----------------------------------------------------------- > > (Updated July 17, 2013, 10:19 p.m.) > > > Review request for hive and Ashutosh Chauhan. > > > Repository: hive-git > > > Description > ------- > > With Dynamic partitioning, Hive would scan default partitions in some cases > even if query excludes it. As part of partition pruning, predicate is > narrowed down to those pieces that involve partition columns only. This > predicate is then evaluated with partition values to determine, if scan > should include those partitions. > But in some cases (like when comparing "_HIVE_DEFAULT_PARTITION_" to numeric > data types) expression evaluation would fail and would return NULL instead of > true/false. In such cases the partition is added to unknown partitions which > is then subsequently scanned. > > This fix avoids scanning default partition if all of the following is true: > a) Hive dynamic partition mode is strict > (hive.exec.dynamic.partition.mode=strict). > b) partition pruning expression failed to evaluate for a given partition. > c) at the least one of the columns in the partition is default partition. > > > Diffs > ----- > > ql/src/java/org/apache/hadoop/hive/ql/optimizer/ppr/PartitionPruner.java > 6a4a360 > ql/src/test/queries/clientpositive/dynamic_partition_skip_default.q > PRE-CREATION > ql/src/test/results/clientpositive/dynamic_partition_skip_default.q.out > PRE-CREATION > > Diff: https://reviews.apache.org/r/12705/diff/ > > > Testing > ------- > > Hive Unit Tests Passed. > > > Thanks, > > John Pullokkaran > >
