[ 
https://issues.apache.org/jira/browse/HIVE-11394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15852364#comment-15852364
 ] 

Hive QA commented on HIVE-11394:
--------------------------------



Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12850877/HIVE-11394.0992.patch

{color:red}ERROR:{color} -1 due to build exiting with an error

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/3363/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/3363/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-3363/

Messages:
{noformat}
**** This message was trimmed, see log for full details ****
error: ql/src/test/results/clientpositive/vector_aggregate_9.q.out: patch does 
not apply
error: patch failed: 
ql/src/test/results/clientpositive/vector_aggregate_without_gby.q.out:31
error: ql/src/test/results/clientpositive/vector_aggregate_without_gby.q.out: 
patch does not apply
error: patch failed: 
ql/src/test/results/clientpositive/vector_between_columns.q.out:61
error: ql/src/test/results/clientpositive/vector_between_columns.q.out: patch 
does not apply
error: patch failed: 
ql/src/test/results/clientpositive/vector_binary_join_groupby.q.out:95
error: ql/src/test/results/clientpositive/vector_binary_join_groupby.q.out: 
patch does not apply
error: patch failed: ql/src/test/results/clientpositive/vector_bround.q.out:32
error: ql/src/test/results/clientpositive/vector_bround.q.out: patch does not 
apply
error: patch failed: ql/src/test/results/clientpositive/vector_bucket.q.out:6
error: ql/src/test/results/clientpositive/vector_bucket.q.out: patch does not 
apply
error: patch failed: 
ql/src/test/results/clientpositive/vector_cast_constant.q.out:95
error: ql/src/test/results/clientpositive/vector_cast_constant.q.out: patch 
does not apply
error: patch failed: ql/src/test/results/clientpositive/vector_char_2.q.out:47
error: ql/src/test/results/clientpositive/vector_char_2.q.out: patch does not 
apply
error: patch failed: ql/src/test/results/clientpositive/vector_char_4.q.out:121
error: ql/src/test/results/clientpositive/vector_char_4.q.out: patch does not 
apply
error: patch failed: 
ql/src/test/results/clientpositive/vector_char_mapjoin1.q.out:124
error: ql/src/test/results/clientpositive/vector_char_mapjoin1.q.out: patch 
does not apply
error: patch failed: 
ql/src/test/results/clientpositive/vector_char_simple.q.out:45
error: ql/src/test/results/clientpositive/vector_char_simple.q.out: patch does 
not apply
error: patch failed: ql/src/test/results/clientpositive/vector_coalesce.q.out:1
error: ql/src/test/results/clientpositive/vector_coalesce.q.out: patch does not 
apply
error: patch failed: 
ql/src/test/results/clientpositive/vector_coalesce_2.q.out:14
error: ql/src/test/results/clientpositive/vector_coalesce_2.q.out: patch does 
not apply
error: patch failed: 
ql/src/test/results/clientpositive/vector_complex_join.q.out:17
error: ql/src/test/results/clientpositive/vector_complex_join.q.out: patch does 
not apply
error: patch failed: ql/src/test/results/clientpositive/vector_count.q.out:43
error: ql/src/test/results/clientpositive/vector_count.q.out: patch does not 
apply
error: patch failed: 
ql/src/test/results/clientpositive/vector_data_types.q.out:95
error: ql/src/test/results/clientpositive/vector_data_types.q.out: patch does 
not apply
error: patch failed: 
ql/src/test/results/clientpositive/vector_decimal_aggregate.q.out:20
error: ql/src/test/results/clientpositive/vector_decimal_aggregate.q.out: patch 
does not apply
error: patch failed: 
ql/src/test/results/clientpositive/vector_decimal_cast.q.out:1
error: ql/src/test/results/clientpositive/vector_decimal_cast.q.out: patch does 
not apply
error: patch failed: 
ql/src/test/results/clientpositive/vector_decimal_expressions.q.out:11
error: ql/src/test/results/clientpositive/vector_decimal_expressions.q.out: 
patch does not apply
error: patch failed: 
ql/src/test/results/clientpositive/vector_decimal_mapjoin.q.out:72
error: ql/src/test/results/clientpositive/vector_decimal_mapjoin.q.out: patch 
does not apply
error: patch failed: 
ql/src/test/results/clientpositive/vector_decimal_math_funcs.q.out:12
error: ql/src/test/results/clientpositive/vector_decimal_math_funcs.q.out: 
patch does not apply
error: patch failed: 
ql/src/test/results/clientpositive/vector_decimal_precision.q.out:545
error: ql/src/test/results/clientpositive/vector_decimal_precision.q.out: patch 
does not apply
error: patch failed: 
ql/src/test/results/clientpositive/vector_decimal_round.q.out:28
error: ql/src/test/results/clientpositive/vector_decimal_round.q.out: patch 
does not apply
error: patch failed: 
ql/src/test/results/clientpositive/vector_decimal_round_2.q.out:24
error: ql/src/test/results/clientpositive/vector_decimal_round_2.q.out: patch 
does not apply
error: patch failed: 
ql/src/test/results/clientpositive/vector_decimal_udf2.q.out:48
error: ql/src/test/results/clientpositive/vector_decimal_udf2.q.out: patch does 
not apply
error: patch failed: 
ql/src/test/results/clientpositive/vector_distinct_2.q.out:101
error: ql/src/test/results/clientpositive/vector_distinct_2.q.out: patch does 
not apply
error: patch failed: ql/src/test/results/clientpositive/vector_elt.q.out:1
error: ql/src/test/results/clientpositive/vector_elt.q.out: patch does not apply
error: patch failed: 
ql/src/test/results/clientpositive/vector_empty_where.q.out:1
error: ql/src/test/results/clientpositive/vector_empty_where.q.out: patch does 
not apply
error: patch failed: ql/src/test/results/clientpositive/vector_groupby4.q.out:18
error: ql/src/test/results/clientpositive/vector_groupby4.q.out: patch does not 
apply
error: patch failed: ql/src/test/results/clientpositive/vector_groupby6.q.out:18
error: ql/src/test/results/clientpositive/vector_groupby6.q.out: patch does not 
apply
error: patch failed: 
ql/src/test/results/clientpositive/vector_groupby_3.q.out:101
error: ql/src/test/results/clientpositive/vector_groupby_3.q.out: patch does 
not apply
error: patch failed: 
ql/src/test/results/clientpositive/vector_groupby_mapjoin.q.out:1
error: ql/src/test/results/clientpositive/vector_groupby_mapjoin.q.out: patch 
does not apply
error: patch failed: 
ql/src/test/results/clientpositive/vector_groupby_reduce.q.out:211
error: ql/src/test/results/clientpositive/vector_groupby_reduce.q.out: patch 
does not apply
error: patch failed: 
ql/src/test/results/clientpositive/vector_grouping_sets.q.out:123
error: ql/src/test/results/clientpositive/vector_grouping_sets.q.out: patch 
does not apply
error: patch failed: ql/src/test/results/clientpositive/vector_if_expr.q.out:1
error: ql/src/test/results/clientpositive/vector_if_expr.q.out: patch does not 
apply
error: patch failed: 
ql/src/test/results/clientpositive/vector_include_no_sel.q.out:163
error: ql/src/test/results/clientpositive/vector_include_no_sel.q.out: patch 
does not apply
error: patch failed: 
ql/src/test/results/clientpositive/vector_interval_1.q.out:38
error: ql/src/test/results/clientpositive/vector_interval_1.q.out: patch does 
not apply
error: patch failed: 
ql/src/test/results/clientpositive/vector_interval_arithmetic.q.out:35
error: ql/src/test/results/clientpositive/vector_interval_arithmetic.q.out: 
patch does not apply
error: patch failed: 
ql/src/test/results/clientpositive/vector_interval_mapjoin.q.out:136
error: ql/src/test/results/clientpositive/vector_interval_mapjoin.q.out: patch 
does not apply
error: patch failed: 
ql/src/test/results/clientpositive/vector_join_part_col_char.q.out:95
error: ql/src/test/results/clientpositive/vector_join_part_col_char.q.out: 
patch does not apply
error: patch failed: 
ql/src/test/results/clientpositive/vector_left_outer_join.q.out:1
error: ql/src/test/results/clientpositive/vector_left_outer_join.q.out: patch 
does not apply
error: patch failed: 
ql/src/test/results/clientpositive/vector_left_outer_join2.q.out:76
error: ql/src/test/results/clientpositive/vector_left_outer_join2.q.out: patch 
does not apply
error: patch failed: 
ql/src/test/results/clientpositive/vector_mapjoin_reduce.q.out:1
error: ql/src/test/results/clientpositive/vector_mapjoin_reduce.q.out: patch 
does not apply
error: patch failed: 
ql/src/test/results/clientpositive/vector_mr_diff_schema_alias.q.out:208
error: ql/src/test/results/clientpositive/vector_mr_diff_schema_alias.q.out: 
patch does not apply
error: patch failed: 
ql/src/test/results/clientpositive/vector_multi_insert.q.out:65
error: ql/src/test/results/clientpositive/vector_multi_insert.q.out: patch does 
not apply
error: patch failed: 
ql/src/test/results/clientpositive/vector_non_constant_in_expr.q.out:1
error: ql/src/test/results/clientpositive/vector_non_constant_in_expr.q.out: 
patch does not apply
error: patch failed: 
ql/src/test/results/clientpositive/vector_non_string_partition.q.out:27
error: ql/src/test/results/clientpositive/vector_non_string_partition.q.out: 
patch does not apply
error: patch failed: 
ql/src/test/results/clientpositive/vector_null_projection.q.out:28
error: ql/src/test/results/clientpositive/vector_null_projection.q.out: patch 
does not apply
error: patch failed: ql/src/test/results/clientpositive/vector_nvl.q.out:1
error: ql/src/test/results/clientpositive/vector_nvl.q.out: patch does not apply
error: patch failed: 
ql/src/test/results/clientpositive/vector_orderby_5.q.out:101
error: ql/src/test/results/clientpositive/vector_orderby_5.q.out: patch does 
not apply
error: patch failed: 
ql/src/test/results/clientpositive/vector_outer_join0.q.out:58
error: ql/src/test/results/clientpositive/vector_outer_join0.q.out: patch does 
not apply
error: patch failed: 
ql/src/test/results/clientpositive/vector_outer_join1.q.out:214
error: ql/src/test/results/clientpositive/vector_outer_join1.q.out: patch does 
not apply
error: patch failed: 
ql/src/test/results/clientpositive/vector_outer_join2.q.out:224
error: ql/src/test/results/clientpositive/vector_outer_join2.q.out: patch does 
not apply
error: patch failed: 
ql/src/test/results/clientpositive/vector_outer_join3.q.out:224
error: ql/src/test/results/clientpositive/vector_outer_join3.q.out: patch does 
not apply
error: patch failed: 
ql/src/test/results/clientpositive/vector_outer_join4.q.out:244
error: ql/src/test/results/clientpositive/vector_outer_join4.q.out: patch does 
not apply
error: patch failed: 
ql/src/test/results/clientpositive/vector_outer_join6.q.out:122
error: ql/src/test/results/clientpositive/vector_outer_join6.q.out: patch does 
not apply
error: patch failed: ql/src/test/results/clientpositive/vector_reduce1.q.out:101
error: ql/src/test/results/clientpositive/vector_reduce1.q.out: patch does not 
apply
error: patch failed: ql/src/test/results/clientpositive/vector_reduce2.q.out:101
error: ql/src/test/results/clientpositive/vector_reduce2.q.out: patch does not 
apply
error: patch failed: ql/src/test/results/clientpositive/vector_reduce3.q.out:101
error: ql/src/test/results/clientpositive/vector_reduce3.q.out: patch does not 
apply
error: patch failed: 
ql/src/test/results/clientpositive/vector_reduce_groupby_decimal.q.out:14
error: ql/src/test/results/clientpositive/vector_reduce_groupby_decimal.q.out: 
patch does not apply
error: patch failed: 
ql/src/test/results/clientpositive/vector_string_concat.q.out:95
error: ql/src/test/results/clientpositive/vector_string_concat.q.out: patch 
does not apply
error: patch failed: 
ql/src/test/results/clientpositive/vector_string_decimal.q.out:38
error: ql/src/test/results/clientpositive/vector_string_decimal.q.out: patch 
does not apply
error: patch failed: 
ql/src/test/results/clientpositive/vector_struct_in.q.out:14
error: ql/src/test/results/clientpositive/vector_struct_in.q.out: patch does 
not apply
error: patch failed: 
ql/src/test/results/clientpositive/vector_tablesample_rows.q.out:1
error: ql/src/test/results/clientpositive/vector_tablesample_rows.q.out: patch 
does not apply
error: patch failed: ql/src/test/results/clientpositive/vector_udf2.q.out:24
error: ql/src/test/results/clientpositive/vector_udf2.q.out: patch does not 
apply
error: patch failed: ql/src/test/results/clientpositive/vector_udf3.q.out:4
error: ql/src/test/results/clientpositive/vector_udf3.q.out: patch does not 
apply
error: patch failed: 
ql/src/test/results/clientpositive/vector_varchar_4.q.out:121
error: ql/src/test/results/clientpositive/vector_varchar_4.q.out: patch does 
not apply
error: patch failed: 
ql/src/test/results/clientpositive/vector_varchar_mapjoin1.q.out:124
error: ql/src/test/results/clientpositive/vector_varchar_mapjoin1.q.out: patch 
does not apply
error: patch failed: 
ql/src/test/results/clientpositive/vector_varchar_simple.q.out:45
error: ql/src/test/results/clientpositive/vector_varchar_simple.q.out: patch 
does not apply
error: patch failed: 
ql/src/test/results/clientpositive/vector_when_case_null.q.out:14
error: ql/src/test/results/clientpositive/vector_when_case_null.q.out: patch 
does not apply
error: patch failed: ql/src/test/results/clientpositive/vectorization_13.q.out:1
error: ql/src/test/results/clientpositive/vectorization_13.q.out: patch does 
not apply
error: patch failed: ql/src/test/results/clientpositive/vectorization_14.q.out:1
error: ql/src/test/results/clientpositive/vectorization_14.q.out: patch does 
not apply
error: patch failed: ql/src/test/results/clientpositive/vectorization_15.q.out:1
error: ql/src/test/results/clientpositive/vectorization_15.q.out: patch does 
not apply
error: patch failed: ql/src/test/results/clientpositive/vectorization_16.q.out:1
error: ql/src/test/results/clientpositive/vectorization_16.q.out: patch does 
not apply
error: patch failed: ql/src/test/results/clientpositive/vectorization_17.q.out:1
error: ql/src/test/results/clientpositive/vectorization_17.q.out: patch does 
not apply
error: patch failed: ql/src/test/results/clientpositive/vectorization_7.q.out:1
error: ql/src/test/results/clientpositive/vectorization_7.q.out: patch does not 
apply
error: patch failed: ql/src/test/results/clientpositive/vectorization_8.q.out:1
error: ql/src/test/results/clientpositive/vectorization_8.q.out: patch does not 
apply
error: patch failed: ql/src/test/results/clientpositive/vectorization_9.q.out:1
error: ql/src/test/results/clientpositive/vectorization_9.q.out: patch does not 
apply
error: patch failed: 
ql/src/test/results/clientpositive/vectorization_decimal_date.q.out:12
error: ql/src/test/results/clientpositive/vectorization_decimal_date.q.out: 
patch does not apply
error: patch failed: 
ql/src/test/results/clientpositive/vectorization_div0.q.out:1
error: ql/src/test/results/clientpositive/vectorization_div0.q.out: patch does 
not apply
error: patch failed: 
ql/src/test/results/clientpositive/vectorization_limit.q.out:1
error: ql/src/test/results/clientpositive/vectorization_limit.q.out: patch does 
not apply
error: patch failed: 
ql/src/test/results/clientpositive/vectorization_offset_limit.q.out:1
error: ql/src/test/results/clientpositive/vectorization_offset_limit.q.out: 
patch does not apply
error: patch failed: 
ql/src/test/results/clientpositive/vectorization_part_project.q.out:46
error: ql/src/test/results/clientpositive/vectorization_part_project.q.out: 
patch does not apply
error: patch failed: 
ql/src/test/results/clientpositive/vectorization_pushdown.q.out:1
error: ql/src/test/results/clientpositive/vectorization_pushdown.q.out: patch 
does not apply
error: patch failed: 
ql/src/test/results/clientpositive/vectorized_bucketmapjoin1.q.out:94
error: ql/src/test/results/clientpositive/vectorized_bucketmapjoin1.q.out: 
patch does not apply
error: patch failed: ql/src/test/results/clientpositive/vectorized_case.q.out:1
error: ql/src/test/results/clientpositive/vectorized_case.q.out: patch does not 
apply
error: patch failed: ql/src/test/results/clientpositive/vectorized_casts.q.out:1
error: ql/src/test/results/clientpositive/vectorized_casts.q.out: patch does 
not apply
error: patch failed: 
ql/src/test/results/clientpositive/vectorized_context.q.out:82
error: ql/src/test/results/clientpositive/vectorized_context.q.out: patch does 
not apply
error: patch failed: 
ql/src/test/results/clientpositive/vectorized_date_funcs.q.out:197
error: ql/src/test/results/clientpositive/vectorized_date_funcs.q.out: patch 
does not apply
error: patch failed: 
ql/src/test/results/clientpositive/vectorized_distinct_gby.q.out:16
error: ql/src/test/results/clientpositive/vectorized_distinct_gby.q.out: patch 
does not apply
error: patch failed: 
ql/src/test/results/clientpositive/vectorized_mapjoin.q.out:1
error: ql/src/test/results/clientpositive/vectorized_mapjoin.q.out: patch does 
not apply
error: patch failed: 
ql/src/test/results/clientpositive/vectorized_mapjoin2.q.out:28
error: ql/src/test/results/clientpositive/vectorized_mapjoin2.q.out: patch does 
not apply
error: patch failed: 
ql/src/test/results/clientpositive/vectorized_math_funcs.q.out:1
error: ql/src/test/results/clientpositive/vectorized_math_funcs.q.out: patch 
does not apply
error: patch failed: 
ql/src/test/results/clientpositive/vectorized_parquet_types.q.out:115
error: ql/src/test/results/clientpositive/vectorized_parquet_types.q.out: patch 
does not apply
error: patch failed: 
ql/src/test/results/clientpositive/vectorized_shufflejoin.q.out:1
error: ql/src/test/results/clientpositive/vectorized_shufflejoin.q.out: patch 
does not apply
error: patch failed: 
ql/src/test/results/clientpositive/vectorized_string_funcs.q.out:1
error: ql/src/test/results/clientpositive/vectorized_string_funcs.q.out: patch 
does not apply
error: patch failed: 
ql/src/test/results/clientpositive/vectorized_timestamp.q.out:17
error: ql/src/test/results/clientpositive/vectorized_timestamp.q.out: patch 
does not apply
error: patch failed: 
ql/src/test/results/clientpositive/vectorized_timestamp_funcs.q.out:63
error: ql/src/test/results/clientpositive/vectorized_timestamp_funcs.q.out: 
patch does not apply
error: patch failed: 
ql/src/test/results/clientpositive/vectorized_timestamp_ints_casts.q.out:1
error: 
ql/src/test/results/clientpositive/vectorized_timestamp_ints_casts.q.out: patch 
does not apply
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12850877 - PreCommit-HIVE-Build

> Enhance EXPLAIN display for vectorization
> -----------------------------------------
>
>                 Key: HIVE-11394
>                 URL: https://issues.apache.org/jira/browse/HIVE-11394
>             Project: Hive
>          Issue Type: Bug
>          Components: Hive
>            Reporter: Matt McCline
>            Assignee: Matt McCline
>            Priority: Critical
>             Fix For: 2.2.0
>
>         Attachments: HIVE-11394.01.patch, HIVE-11394.02.patch, 
> HIVE-11394.03.patch, HIVE-11394.04.patch, HIVE-11394.05.patch, 
> HIVE-11394.06.patch, HIVE-11394.07.patch, HIVE-11394.08.patch, 
> HIVE-11394.091.patch, HIVE-11394.092.patch, HIVE-11394.093.patch, 
> HIVE-11394.094.patch, HIVE-11394.095.patch, HIVE-11394.096.patch, 
> HIVE-11394.097.patch, HIVE-11394.098.patch, HIVE-11394.0991.patch, 
> HIVE-11394.0992.patch, HIVE-11394.099.patch, HIVE-11394.09.patch
>
>
> Add detail to the EXPLAIN output showing why a Map and Reduce work is not 
> vectorized.
> New syntax is: EXPLAIN VECTORIZATION \[ONLY\] 
> \[SUMMARY|OPERATOR|EXPRESSION|DETAIL\]
> The ONLY option suppresses most non-vectorization elements.
> SUMMARY shows vectorization information for the PLAN (is vectorization 
> enabled) and a summary of Map and Reduce work.
> OPERATOR shows vectorization information for operators.  E.g. Filter 
> Vectorization.  It includes all information of SUMMARY, too.
> EXPRESSION shows vectorization information for expressions.  E.g. 
> predicateExpression.  It includes all information of SUMMARY and OPERATOR, 
> too.
> DETAIL shows very vectorization information.
> It includes all information of SUMMARY, OPERATOR, and EXPRESSION too.
> The optional clause defaults are not ONLY and SUMMARY.
> ---------------------------------------------------------------------------------------------------
> Here are some examples:
> EXPLAIN VECTORIZATION example:
> (Note the PLAN VECTORIZATION, Map Vectorization, Reduce Vectorization 
> sections)
> Since SUMMARY is the default, it is the output of EXPLAIN VECTORIZATION 
> SUMMARY.
> Under Reducer 3’s "Reduce Vectorization:" you’ll see
> notVectorizedReason: Aggregation Function UDF avg parameter expression for 
> GROUPBY operator: Data type struct<count:bigint,sum:double,input:int> of 
> Column\[VALUE._col2\] not supported
> For Reducer 2’s "Reduce Vectorization:" you’ll see "groupByVectorOutput:": 
> "false" which says a node has a GROUP BY with an AVG or some other aggregator 
> that outputs a non-PRIMITIVE type (e.g. STRUCT) and all downstream operators 
> are row-mode.  I.e. not vector output.
> If "usesVectorUDFAdaptor:": "false" were true, it would say there was at 
> least one vectorized expression is using VectorUDFAdaptor.
> And, "allNative:": "false" will be true when all operators are native.  
> Today, GROUP BY and FILE SINK are not native.  MAP JOIN and REDUCE SINK are 
> conditionally native.  FILTER and SELECT are native.
> {code}
> PLAN VECTORIZATION:
>   enabled: true
>   enabledConditionsMet: [hive.vectorized.execution.enabled IS true]
> STAGE DEPENDENCIES:
>   Stage-1 is a root stage
>   Stage-0 depends on stages: Stage-1
> STAGE PLANS:
>   Stage: Stage-1
>     Tez
> ...
>       Edges:
>         Reducer 2 <- Map 1 (SIMPLE_EDGE)
>         Reducer 3 <- Reducer 2 (SIMPLE_EDGE)
> ...
>       Vertices:
>         Map 1 
>             Map Operator Tree:
>                 TableScan
>                   alias: alltypesorc
>                   Statistics: Num rows: 12288 Data size: 36696 Basic stats: 
> COMPLETE Column stats: COMPLETE
>                   Select Operator
>                     expressions: cint (type: int)
>                     outputColumnNames: cint
>                     Statistics: Num rows: 12288 Data size: 36696 Basic stats: 
> COMPLETE Column stats: COMPLETE
>                     Group By Operator
>                       keys: cint (type: int)
>                       mode: hash
>                       outputColumnNames: _col0
>                       Statistics: Num rows: 5775 Data size: 17248 Basic 
> stats: COMPLETE Column stats: COMPLETE
>                       Reduce Output Operator
>                         key expressions: _col0 (type: int)
>                         sort order: +
>                         Map-reduce partition columns: _col0 (type: int)
>                         Statistics: Num rows: 5775 Data size: 17248 Basic 
> stats: COMPLETE Column stats: COMPLETE
>             Execution mode: vectorized, llap
>             LLAP IO: all inputs
>             Map Vectorization:
>                 enabled: true
>                 enabledConditionsMet: 
> hive.vectorized.use.vectorized.input.format IS true
>                 groupByVectorOutput: true
>                 inputFileFormats: 
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat
>                 allNative: false
>                 usesVectorUDFAdaptor: false
>                 vectorized: true
>         Reducer 2 
>             Execution mode: vectorized, llap
>             Reduce Vectorization:
>                 enabled: true
>                 enableConditionsMet: hive.vectorized.execution.reduce.enabled 
> IS true, hive.execution.engine tez IN [tez, spark] IS true
>                 groupByVectorOutput: false
>                 allNative: false
>                 usesVectorUDFAdaptor: false
>                 vectorized: true
>             Reduce Operator Tree:
>               Group By Operator
>                 keys: KEY._col0 (type: int)
>                 mode: mergepartial
>                 outputColumnNames: _col0
>                 Statistics: Num rows: 5775 Data size: 17248 Basic stats: 
> COMPLETE Column stats: COMPLETE
>                 Group By Operator
>                   aggregations: sum(_col0), count(_col0), avg(_col0), 
> std(_col0)
>                   mode: hash
>                   outputColumnNames: _col0, _col1, _col2, _col3
>                   Statistics: Num rows: 1 Data size: 172 Basic stats: 
> COMPLETE Column stats: COMPLETE
>                   Reduce Output Operator
>                     sort order: 
>                     Statistics: Num rows: 1 Data size: 172 Basic stats: 
> COMPLETE Column stats: COMPLETE
>                     value expressions: _col0 (type: bigint), _col1 (type: 
> bigint), _col2 (type: struct<count:bigint,sum:double,input:int>), _col3 
> (type: struct<count:bigint,sum:double,variance:double>)
>         Reducer 3 
>             Execution mode: llap
>             Reduce Vectorization:
>                 enabled: true
>                 enableConditionsMet: hive.vectorized.execution.reduce.enabled 
> IS true, hive.execution.engine tez IN [tez, spark] IS true
>                 notVectorizedReason: Aggregation Function UDF avg parameter 
> expression for GROUPBY operator: Data type 
> struct<count:bigint,sum:double,input:int> of Column[VALUE._col2] not supported
>                 vectorized: false
>             Reduce Operator Tree:
>               Group By Operator
>                 aggregations: sum(VALUE._col0), count(VALUE._col1), 
> avg(VALUE._col2), std(VALUE._col3)
>                 mode: mergepartial
>                 outputColumnNames: _col0, _col1, _col2, _col3
>                 Statistics: Num rows: 1 Data size: 32 Basic stats: COMPLETE 
> Column stats: COMPLETE
>                 File Output Operator
>                   compressed: false
>                   Statistics: Num rows: 1 Data size: 32 Basic stats: COMPLETE 
> Column stats: COMPLETE
>                   table:
>                       input format: 
> org.apache.hadoop.mapred.SequenceFileInputFormat
>                       output format: 
> org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
>                       serde: 
> org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
>   Stage: Stage-0
>     Fetch Operator
>       limit: -1
>       Processor Tree:
>         ListSink 
> {code}
> EXPLAIN VECTORIZATION OPERATOR
> Notice the added  TableScan Vectorization, Select Vectorization, Group By 
> Vectorization, Map Join Vectorizatin, Reduce Sink Vectorization sections in 
> this example.
> Notice the nativeConditionsMet detail on why Reduce Vectorization is native.
> {code}
> PLAN VECTORIZATION:
>   enabled: true
>   enabledConditionsMet: [hive.vectorized.execution.enabled IS true]
> STAGE DEPENDENCIES:
>   Stage-1 is a root stage
>   Stage-0 depends on stages: Stage-1
> STAGE PLANS:
>   Stage: Stage-1
>     Tez
> #### A masked pattern was here ####
>       Edges:
>         Map 2 <- Map 1 (BROADCAST_EDGE)
>         Reducer 3 <- Map 2 (SIMPLE_EDGE)
> #### A masked pattern was here ####
>       Vertices:
>         Map 1 
>             Map Operator Tree:
>                 TableScan
>                   alias: a
>                   Statistics: Num rows: 3 Data size: 294 Basic stats: 
> COMPLETE Column stats: NONE
>                   TableScan Vectorization:
>                       native: true
>                       projectedOutputColumns: [0, 1]
>                   Filter Operator
>                     Filter Vectorization:
>                         className: VectorFilterOperator
>                         native: true
> predicate: c2 is not null (type: boolean)
>                     Statistics: Num rows: 3 Data size: 294 Basic stats: 
> COMPLETE Column stats: NONE
>                     Select Operator
>                       expressions: c1 (type: int), c2 (type: char(10))
>                       outputColumnNames: _col0, _col1
>                       Select Vectorization:
>                           className: VectorSelectOperator
>                           native: true
>                           projectedOutputColumns: [0, 1]
>                       Statistics: Num rows: 3 Data size: 294 Basic stats: 
> COMPLETE Column stats: NONE
>                       Reduce Output Operator
>                         key expressions: _col1 (type: char(20))
>                         sort order: +
>                         Map-reduce partition columns: _col1 (type: char(20))
>                         Reduce Sink Vectorization:
>                             className: VectorReduceSinkStringOperator
>                             native: true
>                             nativeConditionsMet: 
> hive.vectorized.execution.reducesink.new.enabled IS true, 
> hive.execution.engine tez IN [tez, spark] IS true, Not ACID UPDATE or DELETE 
> IS true, No buckets IS true, No TopN IS true, Uniform Hash IS true, No 
> DISTINCT columns IS true, BinarySortableSerDe for keys IS true, 
> LazyBinarySerDe for values IS true
>                         Statistics: Num rows: 3 Data size: 294 Basic stats: 
> COMPLETE Column stats: NONE
>                         value expressions: _col0 (type: int)
>             Execution mode: vectorized, llap
>             LLAP IO: all inputs
>             Map Vectorization:
>                 enabled: true
>                 enabledConditionsMet: 
> hive.vectorized.use.vectorized.input.format IS true
>                 groupByVectorOutput: true
>                 inputFileFormats: 
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat
>                 allNative: true
>                 usesVectorUDFAdaptor: false
>                 vectorized: true
>         Map 2 
>             Map Operator Tree:
>                 TableScan
>                   alias: b
>                   Statistics: Num rows: 3 Data size: 324 Basic stats: 
> COMPLETE Column stats: NONE
>                   TableScan Vectorization:
>                       native: true
>                       projectedOutputColumns: [0, 1]
>                   Filter Operator
>                     Filter Vectorization:
>                         className: VectorFilterOperator
>                         native: true
> predicate: c2 is not null (type: boolean)
>                     Statistics: Num rows: 3 Data size: 324 Basic stats: 
> COMPLETE Column stats: NONE
>                     Select Operator
>                       expressions: c1 (type: int), c2 (type: char(20))
>                       outputColumnNames: _col0, _col1
>                       Select Vectorization:
>                           className: VectorSelectOperator
>                           native: true
>                           projectedOutputColumns: [0, 1]
>                       Statistics: Num rows: 3 Data size: 324 Basic stats: 
> COMPLETE Column stats: NONE
>                       Map Join Operator
>                         condition map:
>                              Inner Join 0 to 1
>                         keys:
>                           0 _col1 (type: char(20))
>                           1 _col1 (type: char(20))
>                         Map Join Vectorization:
>                             className: VectorMapJoinInnerStringOperator
>                             native: true
>                             nativeConditionsMet: 
> hive.vectorized.execution.mapjoin.native.enabled IS true, 
> hive.execution.engine tez IN [tez, spark] IS true, One MapJoin Condition IS 
> true, No nullsafe IS true, Supports Key Types IS true, Not empty key IS true, 
> When Fast Hash Table, then requires no Hybrid Hash Join IS true, Small table 
> vectorizes IS true
>                         outputColumnNames: _col0, _col1, _col2, _col3
>                         input vertices:
>                           0 Map 1
>                         Statistics: Num rows: 3 Data size: 323 Basic stats: 
> COMPLETE Column stats: NONE
>                         Reduce Output Operator
>                           key expressions: _col0 (type: int)
>                           sort order: +
>                           Reduce Sink Vectorization:
>                               className: VectorReduceSinkOperator
>                               native: false
>                               nativeConditionsMet: 
> hive.vectorized.execution.reducesink.new.enabled IS true, 
> hive.execution.engine tez IN [tez, spark] IS true, Not ACID UPDATE or DELETE 
> IS true, No buckets IS true, No TopN IS true, No DISTINCT columns IS true, 
> BinarySortableSerDe for keys IS true, LazyBinarySerDe for values IS true
>                               nativeConditionsNotMet: Uniform Hash IS false
>                           Statistics: Num rows: 3 Data size: 323 Basic stats: 
> COMPLETE Column stats: NONE
>                           value expressions: _col1 (type: char(10)), _col2 
> (type: int), _col3 (type: char(20))
>             Execution mode: vectorized, llap
>             LLAP IO: all inputs
>             Map Vectorization:
>                 enabled: true
>                 enabledConditionsMet: 
> hive.vectorized.use.vectorized.input.format IS true
>                 groupByVectorOutput: true
>                 inputFileFormats: 
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat
>                 allNative: false
>                 usesVectorUDFAdaptor: false
>                 vectorized: true
>         Reducer 3 
>             Execution mode: vectorized, llap
>             Reduce Vectorization:
>                 enabled: true
>                 enableConditionsMet: hive.vectorized.execution.reduce.enabled 
> IS true, hive.execution.engine tez IN [tez, spark] IS true
>                 groupByVectorOutput: true
>                 allNative: false
>                 usesVectorUDFAdaptor: false
>                 vectorized: true
>             Reduce Operator Tree:
>               Select Operator
>                 expressions: KEY.reducesinkkey0 (type: int), VALUE._col0 
> (type: char(10)), VALUE._col1 (type: int), VALUE._col2 (type: char(20))
>                 outputColumnNames: _col0, _col1, _col2, _col3
>                 Select Vectorization:
>                     className: VectorSelectOperator
>                     native: true
>                     projectedOutputColumns: [0, 1, 2, 3]
>                 Statistics: Num rows: 3 Data size: 323 Basic stats: COMPLETE 
> Column stats: NONE
>                 File Output Operator
>                   compressed: false
>                   File Sink Vectorization:
>                       className: VectorFileSinkOperator
>                       native: false
>                   Statistics: Num rows: 3 Data size: 323 Basic stats: 
> COMPLETE Column stats: NONE
>                   table:
>                       input format: 
> org.apache.hadoop.mapred.SequenceFileInputFormat
>                       output format: 
> org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
>                       serde: 
> org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
>   Stage: Stage-0
>     Fetch Operator
>       limit: -1
>       Processor Tree:
>         ListSink
>  {code}
> EXPLAIN VECTORIZATION EXPRESSION
> Notice the predicateExpression in this example.
> {code}
> PLAN VECTORIZATION:
>   enabled: true
>   enabledConditionsMet: [hive.vectorized.execution.enabled IS true]
> STAGE DEPENDENCIES:
>   Stage-1 is a root stage
>   Stage-0 depends on stages: Stage-1
> STAGE PLANS:
>   Stage: Stage-1
>     Tez
> #### A masked pattern was here ####
>       Edges:
>         Reducer 2 <- Map 1 (SIMPLE_EDGE)
> #### A masked pattern was here ####
>       Vertices:
>         Map 1 
>             Map Operator Tree:
>                 TableScan
>                   alias: vector_interval_2
>                   Statistics: Num rows: 2 Data size: 788 Basic stats: 
> COMPLETE Column stats: NONE
>                   TableScan Vectorization:
>                       native: true
>                       projectedOutputColumns: [0, 1, 2, 3, 4, 5]
>                   Filter Operator
>                     Filter Vectorization:
>                         className: VectorFilterOperator
>                         native: true
>                         predicateExpression: FilterExprAndExpr(children: 
> FilterTimestampScalarEqualTimestampColumn(val 2001-01-01 01:02:03.0, col 
> 6)(children: DateColAddIntervalDayTimeScalar(col 1, val 0 01:02:03.000000000) 
> -> 6:timestamp) -> boolean, FilterTimestampScalarNotEqualTimestampColumn(val 
> 2001-01-01 01:02:03.0, col 6)(children: DateColAddIntervalDayTimeScalar(col 
> 1, val 0 01:02:04.000000000) -> 6:timestamp) -> boolean, 
> FilterTimestampScalarLessEqualTimestampColumn(val 2001-01-01 01:02:03.0, col 
> 6)(children: DateColAddIntervalDayTimeScalar(col 1, val 0 01:02:03.000000000) 
> -> 6:timestamp) -> boolean, FilterTimestampScalarLessTimestampColumn(val 
> 2001-01-01 01:02:03.0, col 6)(children: DateColAddIntervalDayTimeScalar(col 
> 1, val 0 01:02:04.000000000) -> 6:timestamp) -> boolean, 
> FilterTimestampScalarGreaterEqualTimestampColumn(val 2001-01-01 01:02:03.0, 
> col 6)(children: DateColSubtractIntervalDayTimeScalar(col 1, val 0 
> 01:02:03.000000000) -> 6:timestamp) -> boolean, 
> FilterTimestampScalarGreaterTimestampColumn(val 2001-01-01 01:02:03.0, col 
> 6)(children: DateColSubtractIntervalDayTimeScalar(col 1, val 0 
> 01:02:04.000000000) -> 6:timestamp) -> boolean, 
> FilterTimestampColEqualTimestampScalar(col 6, val 2001-01-01 
> 01:02:03.0)(children: DateColAddIntervalDayTimeScalar(col 1, val 0 
> 01:02:03.000000000) -> 6:timestamp) -> boolean, 
> FilterTimestampColNotEqualTimestampScalar(col 6, val 2001-01-01 
> 01:02:03.0)(children: DateColAddIntervalDayTimeScalar(col 1, val 0 
> 01:02:04.000000000) -> 6:timestamp) -> boolean, 
> FilterTimestampColGreaterEqualTimestampScalar(col 6, val 2001-01-01 
> 01:02:03.0)(children: DateColAddIntervalDayTimeScalar(col 1, val 0 
> 01:02:03.000000000) -> 6:timestamp) -> boolean, 
> FilterTimestampColGreaterTimestampScalar(col 6, val 2001-01-01 
> 01:02:03.0)(children: DateColAddIntervalDayTimeScalar(col 1, val 0 
> 01:02:04.000000000) -> 6:timestamp) -> boolean, 
> FilterTimestampColLessEqualTimestampScalar(col 6, val 2001-01-01 
> 01:02:03.0)(children: DateColSubtractIntervalDayTimeScalar(col 1, val 0 
> 01:02:03.000000000) -> 6:timestamp) -> boolean, 
> FilterTimestampColLessTimestampScalar(col 6, val 2001-01-01 
> 01:02:03.0)(children: DateColSubtractIntervalDayTimeScalar(col 1, val 0 
> 01:02:04.000000000) -> 6:timestamp) -> boolean, 
> FilterTimestampColEqualTimestampColumn(col 0, col 6)(children: 
> DateColAddIntervalDayTimeScalar(col 1, val 0 01:02:03.000000000) -> 
> 6:timestamp) -> boolean, FilterTimestampColNotEqualTimestampColumn(col 0, col 
> 6)(children: DateColAddIntervalDayTimeScalar(col 1, val 0 01:02:04.000000000) 
> -> 6:timestamp) -> boolean, FilterTimestampColLessEqualTimestampColumn(col 0, 
> col 6)(children: DateColAddIntervalDayTimeScalar(col 1, val 0 
> 01:02:03.000000000) -> 6:timestamp) -> boolean, 
> FilterTimestampColLessTimestampColumn(col 0, col 6)(children: 
> DateColAddIntervalDayTimeScalar(col 1, val 0 01:02:04.000000000) -> 
> 6:timestamp) -> boolean, FilterTimestampColGreaterEqualTimestampColumn(col 0, 
> col 6)(children: DateColSubtractIntervalDayTimeScalar(col 1, val 0 
> 01:02:03.000000000) -> 6:timestamp) -> boolean, 
> FilterTimestampColGreaterTimestampColumn(col 0, col 6)(children: 
> DateColSubtractIntervalDayTimeScalar(col 1, val 0 01:02:04.000000000) -> 
> 6:timestamp) -> boolean) -> boolean
>                     predicate: ((2001-01-01 01:02:03.0 = (dt + 0 
> 01:02:03.000000000)) and (2001-01-01 01:02:03.0 <> (dt + 0 
> 01:02:04.000000000)) and (2001-01-01 01:02:03.0 <= (dt + 0 
> 01:02:03.000000000)) and (2001-01-01 01:02:03.0 < (dt + 0 
> 01:02:04.000000000)) and (2001-01-01 01:02:03.0 >= (dt - 0 
> 01:02:03.000000000)) and (2001-01-01 01:02:03.0 > (dt - 0 
> 01:02:04.000000000)) and ((dt + 0 01:02:03.000000000) = 2001-01-01 
> 01:02:03.0) and ((dt + 0 01:02:04.000000000) <> 2001-01-01 01:02:03.0) and 
> ((dt + 0 01:02:03.000000000) >= 2001-01-01 01:02:03.0) and ((dt + 0 
> 01:02:04.000000000) > 2001-01-01 01:02:03.0) and ((dt - 0 01:02:03.000000000) 
> <= 2001-01-01 01:02:03.0) and ((dt - 0 01:02:04.000000000) < 2001-01-01 
> 01:02:03.0) and (ts = (dt + 0 01:02:03.000000000)) and (ts <> (dt + 0 
> 01:02:04.000000000)) and (ts <= (dt + 0 01:02:03.000000000)) and (ts < (dt + 
> 0 01:02:04.000000000)) and (ts >= (dt - 0 01:02:03.000000000)) and (ts > (dt 
> - 0 01:02:04.000000000))) (type: boolean)
>                     Statistics: Num rows: 1 Data size: 394 Basic stats: 
> COMPLETE Column stats: NONE
>                     Select Operator
>                       expressions: ts (type: timestamp)
>                       outputColumnNames: _col0
>                       Select Vectorization:
>                           className: VectorSelectOperator
>                           native: true
>                           projectedOutputColumns: [0]
>                       Statistics: Num rows: 1 Data size: 394 Basic stats: 
> COMPLETE Column stats: NONE
>                       Reduce Output Operator
>                         key expressions: _col0 (type: timestamp)
>                         sort order: +
>                         Reduce Sink Vectorization:
>                             className: VectorReduceSinkOperator
>                             native: false
>                             nativeConditionsMet: 
> hive.vectorized.execution.reducesink.new.enabled IS true, 
> hive.execution.engine tez IN [tez, spark] IS true, Not ACID UPDATE or DELETE 
> IS true, No buckets IS true, No TopN IS true, No DISTINCT columns IS true, 
> BinarySortableSerDe for keys IS true, LazyBinarySerDe for values IS true
>                             nativeConditionsNotMet: Uniform Hash IS false
>                         Statistics: Num rows: 1 Data size: 394 Basic stats: 
> COMPLETE Column stats: NONE
>             Execution mode: vectorized, llap
>             LLAP IO: all inputs
>             Map Vectorization:
>                 enabled: true
>                 enabledConditionsMet: 
> hive.vectorized.use.vectorized.input.format IS true
>                 groupByVectorOutput: true
>                 inputFileFormats: 
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat
>                 allNative: false
>                 usesVectorUDFAdaptor: false
>                 vectorized: true
>         Reducer 2 
> ... 
> {code}
> The standard @Explain Annotation Type is used.  A new 'vectorization' 
> annotation marks each new class and method.
> Works for FORMATTED, like other non-vectorization EXPLAIN variations.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Reply via email to