Hey,
I'm trying to understand a hive explain-plan I generated and understand
what's what. I couldn't find any documentation as this
<https://cwiki.apache.org/confluence/display/Hive/LanguageManual+Explain>is
too high-level
basically, the query is an equi-join of 8 tables on the same 2 keys:
select * from c
left outer join i1 on c.PURSWAY_ID1 = i1.PURSWAY_ID1 and c.PURSWAY_ID2 =
i1.PURSWAY_ID2
left outer join i2 on c.PURSWAY_ID1 = i2.PURSWAY_ID1 and c.PURSWAY_ID2 =
i2.PURSWAY_ID2
...
left outer join i7 on c.PURSWAY_ID1 = i7.PURSWAY_ID1 and c.PURSWAY_ID2 =
i7.PURSWAY_ID2
The explain plan at the end of this mail, with parts I need clarification on
in red.
Questions:
1. what is the meaning of the "Map Reduce" line at the beginning?
2. is Select Operator a map-side only operation? if so, why is the reduce
output operator indented inside it?
3. what is the meaning of the "sort order: ++" line?
4. Join Operator - what is the meaning of condition map and condition
expressions?
5. meaning of handleSkewJoin: false
6. where can I find the temporary output files of each map operation?
I'll be grateful for every bit of help :)
Stage: Stage-1
Map Reduce
Alias -> Map Operator Tree:
c:ido_standard_connections
TableScan
alias: ido_standard_connections
Select Operator
expressions:
expr: pursway_id1
expr: pursway_id2
outputColumnNames: _col0, _col1
Reduce Output Operator
key expressions:
expr: _col0
expr: _col1
sort order: ++
Map-reduce partition columns:
expr: _col0
expr: _col1
tag: 0
value expressions:
expr: _col0
expr: _col1
i1
TableScan
alias: i1
Reduce Output Operator
key expressions:
expr: pursway_id1
expr: pursway_id2
sort order: ++
Map-reduce partition columns:
expr: pursway_id1
expr: pursway_id2
tag: 3
value expressions:
expr: ind_products_cnt
expr: ind_products_min_diff_days
expr: ind_products_expectancy
...
Reduce Operator Tree:
Join Operator
condition map:
Left Outer Join0 to 1
Left Outer Join0 to 2
...
Left Outer Join0 to 9
condition expressions:
0 {VALUE._col0} {VALUE._col1}
1 {VALUE._col2}
2 {VALUE._col2}
3 {VALUE._col2} {VALUE._col3} {VALUE._col4}
4 {VALUE._col2} {VALUE._col3} {VALUE._col4}
5 {VALUE._col2} {VALUE._col3} {VALUE._col4}
6 {VALUE._col2} {VALUE._col3} {VALUE._col4}
7 {VALUE._col2} {VALUE._col3} {VALUE._col4}
8 {VALUE._col2} {VALUE._col3} {VALUE._col4}
9 {VALUE._col2} {VALUE._col3} {VALUE._col4}
handleSkewJoin: false
outputColumnNames: _col0, _col1, _col4, _col9 ... _col58
Select Operator
expressions:
expr: _col0
expr: _col1
...
expr: _col58
outputColumnNames: _col0, _col1, _col2 ... _col24
File Output Operator
compressed: false
GlobalTableId: 0
table:
input format: org.apache.hadoop.mapred.TextInputFormat
output format:
org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
