Hello ,
Trying to benchmark with Hive on Tez causes the following error. Admittedly
these are some very large looking records .. the same job runs fine on MR2.
I'v attached the query explain tree. It fails in the very last reducer
phase ..
*Execution:*
--------------------------------------------------------------------------------
VERTICES STATUS TOTAL COMPLETED RUNNING PENDING FAILED
KILLED
--------------------------------------------------------------------------------
Map 1 .......... SUCCEEDED 477 477 0 0 0
0
Reducer 10 ..... SUCCEEDED 250 250 0 0 0
0
Reducer 11 ..... SUCCEEDED 250 250 0 0 1
0
Reducer 12 ..... SUCCEEDED 250 250 0 0 1
0
Reducer 13 ..... SUCCEEDED 250 250 0 0 1
0
Reducer 14 ..... SUCCEEDED 250 250 0 0 0
0
Reducer 15 ..... SUCCEEDED 250 250 0 0 0
0
Reducer 16 ... KILLED 250 187 0 63 0
63
Reducer 17 FAILED 250 0 0 250 62
249
Reducer 2 ...... SUCCEEDED 250 250 0 0 1
0
Reducer 3 ...... SUCCEEDED 250 250 0 0 0
0
Reducer 4 ...... SUCCEEDED 250 250 0 0 0
0
Reducer 5 ...... SUCCEEDED 250 250 0 0 0
0
Reducer 6 ...... SUCCEEDED 250 250 0 0 1
0
Reducer 7 ...... SUCCEEDED 250 250 0 0 0
0
Reducer 8 ...... SUCCEEDED 250 250 0 0 0
0
Reducer 9 ...... SUCCEEDED 250 250 0 0 0
0
--------------------------------------------------------------------------------
VERTICES: 15/17 [========================>>--] 93% ELAPSED TIME:
17600.11 s
--------------------------------------------------------------------------------
*Error: *
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
org.apache.tez.runtime.library.common.sort.impl.ExternalSorter$MapBufferTooSmallException:
Record too large for in-memory buffer. Exceeded buffer overflow limit,
bufferOverflowRecursion=2, bufferList.size=1, blockSize=268435456
at
org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.processOp(ReduceSinkOperator.java:397)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:815)
at
org.apache.hadoop.hive.ql.exec.FilterOperator.processOp(FilterOperator.java:120)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:815)
at
org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:95)
at
org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:157)
at
org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:497)
... 18 more
Caused by:
org.apache.tez.runtime.library.common.sort.impl.ExternalSorter$MapBufferTooSmallException:
Record too large for in-memory buffer. Exceeded buffer overflow limit,
bufferOverflowRecursion=2, bufferList.size=1, blockSize=268435456
at
org.apache.tez.runtime.library.common.sort.impl.PipelinedSorter.collect(PipelinedSorter.java:315)
at
org.apache.tez.runtime.library.common.sort.impl.PipelinedSorter.collect(PipelinedSorter.java:320)
at
org.apache.tez.runtime.library.common.sort.impl.PipelinedSorter.write(PipelinedSorter.java:272)
at
org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput$1.write(OrderedPartitionedKVOutput.java:164)
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor$TezKVOutputCollector.collect(TezProcessor.java:211)
at
org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.collect(ReduceSinkOperator.java:534)
at
org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.processOp(ReduceSinkOperator.java:380)
... 24 more
-Gautam.
OK
STAGE DEPENDENCIES:
Stage-1 is a root stage
Stage-2 depends on stages: Stage-1
Stage-3 depends on stages: Stage-2
Stage-4 depends on stages: Stage-3
Stage-5 depends on stages: Stage-4
Stage-6 depends on stages: Stage-5
Stage-7 depends on stages: Stage-6
Stage-8 depends on stages: Stage-7
Stage-9 depends on stages: Stage-8
Stage-10 depends on stages: Stage-9
Stage-11 depends on stages: Stage-10
Stage-12 depends on stages: Stage-11
Stage-13 depends on stages: Stage-12
Stage-14 depends on stages: Stage-13
Stage-15 depends on stages: Stage-14
Stage-16 depends on stages: Stage-15
Stage-0 depends on stages: Stage-16
STAGE PLANS:
Stage: Stage-1
Map Reduce
Map Operator Tree:
TableScan
alias: upsight_clean_data
Statistics: Num rows: 434752932 Data size: 443448005886 Basic
stats: COMPLETE Column stats: NONE
Filter Operator
predicate: (msg_type like 'pub.%') (type: boolean)
Statistics: Num rows: 217376466 Data size: 221724002943 Basic
stats: COMPLETE Column stats: NONE
Reduce Output Operator
key expressions: value['app_id'] (type: string), value['type']
(type: string), (value['ts'] % 1000) (type: double), value['app_id'] (type:
string), value['type'] (type: string), (value['ts'] % 1000) (type: double)
sort order: ++++++
Map-reduce partition columns: value['app_id'] (type: string),
value['type'] (type: string), (value['ts'] % 1000) (type: double)
Statistics: Num rows: 217376466 Data size: 221724002943 Basic
stats: COMPLETE Column stats: NONE
value expressions: msg_type (type: string), value (type:
map<string,string>)
Reduce Operator Tree:
Select Operator
expressions: VALUE._col0 (type: string), VALUE._col1 (type:
map<string,string>)
outputColumnNames: _col0, _col1
Statistics: Num rows: 217376466 Data size: 221724002943 Basic stats:
COMPLETE Column stats: NONE
PTF Operator
Statistics: Num rows: 217376466 Data size: 221724002943 Basic
stats: COMPLETE Column stats: NONE
Select Operator
expressions: _col1['meta.appId'] (type: string), _col0 (type:
string), _col1['meta.userId'] (type: string), _col1['session_num'] (type:
string), _col1 (type: map<string,string>), _wcol0 (type: bigint), _wcol1 (type:
int), (_col1['ts'] % 1000) (type: double)
outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5,
_col6, _col7
Statistics: Num rows: 217376466 Data size: 221724002943 Basic
stats: COMPLETE Column stats: NONE
File Output Operator
compressed: true
table:
input format:
org.apache.hadoop.mapred.SequenceFileInputFormat
output format:
org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
serde:
org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
Stage: Stage-2
Map Reduce
Map Operator Tree:
TableScan
Reduce Output Operator
key expressions: _col1 (type: string), _col0 (type: string),
_col6 (type: int), _col1 (type: string), _col0 (type: string), _col6 (type: int)
sort order: ++++++
Map-reduce partition columns: _col1 (type: string), _col0 (type:
string), _col6 (type: int)
Statistics: Num rows: 217376466 Data size: 221724002943 Basic
stats: COMPLETE Column stats: NONE
value expressions: _col2 (type: string), _col3 (type: string),
_col4 (type: map<string,string>), _col5 (type: bigint), _col7 (type: double)
Reduce Operator Tree:
Select Operator
expressions: KEY.reducesinkkey1 (type: string), KEY.reducesinkkey0
(type: string), VALUE._col0 (type: string), VALUE._col1 (type: string),
VALUE._col2 (type: map<string,string>), VALUE._col3 (type: bigint),
KEY.reducesinkkey2 (type: int), VALUE._col4 (type: double)
outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6,
_col7
Statistics: Num rows: 217376466 Data size: 221724002943 Basic stats:
COMPLETE Column stats: NONE
PTF Operator
Statistics: Num rows: 217376466 Data size: 221724002943 Basic
stats: COMPLETE Column stats: NONE
Select Operator
expressions: _col0 (type: string), _col1 (type: string), _col2
(type: string), _col3 (type: string), _col4 (type: map<string,string>), _col6
(type: int), _col7 (type: double), _wcol0 (type: bigint)
outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5,
_col6, _col7
Statistics: Num rows: 217376466 Data size: 221724002943 Basic
stats: COMPLETE Column stats: NONE
File Output Operator
compressed: true
table:
input format:
org.apache.hadoop.mapred.SequenceFileInputFormat
output format:
org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
serde:
org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
Stage: Stage-3
Map Reduce
Map Operator Tree:
TableScan
Reduce Output Operator
key expressions: _col6 (type: double), _col0 (type: string),
_col1 (type: string), _col6 (type: double), _col0 (type: string), _col1 (type:
string)
sort order: ++++++
Map-reduce partition columns: _col6 (type: double), _col0 (type:
string), _col1 (type: string)
Statistics: Num rows: 217376466 Data size: 221724002943 Basic
stats: COMPLETE Column stats: NONE
value expressions: _col2 (type: string), _col3 (type: string),
_col4 (type: map<string,string>), _col5 (type: int), _col7 (type: bigint)
Reduce Operator Tree:
Select Operator
expressions: KEY.reducesinkkey1 (type: string), KEY.reducesinkkey2
(type: string), VALUE._col0 (type: string), VALUE._col1 (type: string),
VALUE._col2 (type: map<string,string>), VALUE._col3 (type: int),
KEY.reducesinkkey0 (type: double), VALUE._col4 (type: bigint)
outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6,
_col7
Statistics: Num rows: 217376466 Data size: 221724002943 Basic stats:
COMPLETE Column stats: NONE
PTF Operator
Statistics: Num rows: 217376466 Data size: 221724002943 Basic
stats: COMPLETE Column stats: NONE
Select Operator
expressions: _col0 (type: string), _col1 (type: string), _col2
(type: string), _col3 (type: string), _col4 (type: map<string,string>), _col6
(type: double), _col5 (type: int), _wcol0 (type: bigint)
outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5,
_col6, _col7
Statistics: Num rows: 217376466 Data size: 221724002943 Basic
stats: COMPLETE Column stats: NONE
File Output Operator
compressed: true
table:
input format:
org.apache.hadoop.mapred.SequenceFileInputFormat
output format:
org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
serde:
org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
Stage: Stage-4
Map Reduce
Map Operator Tree:
TableScan
Reduce Output Operator
key expressions: _col0 (type: string), _col6 (type: int), _col7
(type: bigint), _col1 (type: string)
sort order: ++-+
Map-reduce partition columns: _col0 (type: string), _col6 (type:
int)
Statistics: Num rows: 217376466 Data size: 221724002943 Basic
stats: COMPLETE Column stats: NONE
value expressions: _col2 (type: string), _col3 (type: string),
_col4 (type: map<string,string>), _col5 (type: double)
Reduce Operator Tree:
Select Operator
expressions: KEY.reducesinkkey0 (type: string), KEY.reducesinkkey3
(type: string), VALUE._col0 (type: string), VALUE._col1 (type: string),
VALUE._col2 (type: map<string,string>), VALUE._col3 (type: double),
KEY.reducesinkkey1 (type: int), KEY.reducesinkkey2 (type: bigint)
outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6,
_col7
Statistics: Num rows: 217376466 Data size: 221724002943 Basic stats:
COMPLETE Column stats: NONE
PTF Operator
Statistics: Num rows: 217376466 Data size: 221724002943 Basic
stats: COMPLETE Column stats: NONE
Select Operator
expressions: _col0 (type: string), _col1 (type: string), _col2
(type: string), _col3 (type: string), _col4 (type: map<string,string>), _col5
(type: double), _col7 (type: bigint), _col6 (type: int), _wcol0 (type: int)
outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5,
_col6, _col7, _col8
Statistics: Num rows: 217376466 Data size: 221724002943 Basic
stats: COMPLETE Column stats: NONE
File Output Operator
compressed: true
table:
input format:
org.apache.hadoop.mapred.SequenceFileInputFormat
output format:
org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
serde:
org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
Stage: Stage-5
Map Reduce
Map Operator Tree:
TableScan
Reduce Output Operator
key expressions: _col5 (type: double), _col0 (type: string),
_col1 (type: string), _col5 (type: double), _col0 (type: string), _col1 (type:
string)
sort order: ++++++
Map-reduce partition columns: _col5 (type: double), _col0 (type:
string), _col1 (type: string)
Statistics: Num rows: 217376466 Data size: 221724002943 Basic
stats: COMPLETE Column stats: NONE
value expressions: _col2 (type: string), _col3 (type: string),
_col4 (type: map<string,string>), _col6 (type: bigint), _col7 (type: int),
_col8 (type: int)
Reduce Operator Tree:
Select Operator
expressions: KEY.reducesinkkey1 (type: string), KEY.reducesinkkey2
(type: string), VALUE._col0 (type: string), VALUE._col1 (type: string),
VALUE._col2 (type: map<string,string>), KEY.reducesinkkey0 (type: double),
VALUE._col3 (type: bigint), VALUE._col4 (type: int), VALUE._col5 (type: int)
outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6,
_col7, _col8
Statistics: Num rows: 217376466 Data size: 221724002943 Basic stats:
COMPLETE Column stats: NONE
PTF Operator
Statistics: Num rows: 217376466 Data size: 221724002943 Basic
stats: COMPLETE Column stats: NONE
Filter Operator
predicate: (_wcol0 <= 150) (type: boolean)
Statistics: Num rows: 72458822 Data size: 73908000981 Basic
stats: COMPLETE Column stats: NONE
Select Operator
expressions: _col0 (type: string), _col1 (type: string), _col2
(type: string), _col3 (type: string), _col5 (type: double), _col6 (type:
bigint), _col4 (type: map<string,string>)
outputColumnNames: _col0, _col1, _col2, _col3, _col5, _col6,
_col4
Statistics: Num rows: 72458822 Data size: 73908000981 Basic
stats: COMPLETE Column stats: NONE
Lateral View Forward
Statistics: Num rows: 72458822 Data size: 73908000981 Basic
stats: COMPLETE Column stats: NONE
Select Operator
expressions: _col0 (type: string), _col1 (type: string),
_col2 (type: string), _col3 (type: string), _col5 (type: double), _col6 (type:
bigint), _col4 (type: map<string,string>)
outputColumnNames: _col0, _col1, _col2, _col3, _col5,
_col6, _col4
Statistics: Num rows: 72458822 Data size: 73908000981 Basic
stats: COMPLETE Column stats: NONE
Lateral View Join Operator
outputColumnNames: _col0, _col1, _col2, _col3, _col5,
_col6, _col4, _col8
Statistics: Num rows: 144917644 Data size: 147816001962
Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: _col0 (type: string), _col1 (type:
string), _col2 (type: string), _col3 (type: string), _col5 (type: double),
_col6 (type: bigint), _col8 (type: string), if((_col8 = 'standard'),
to_json(map('Country':_col4['location.country_ip'],'App
Version':_col4['app.version'],'OS Version':_col4['device.os_version'],'Device
Type':_col4['device.type'],'Device
Manufacturer':_col4['device.manufacturer'],'Connection
Type':_col4['device.connection'],'Device
Carrier':_col4['device.carrier'],'Language':_col4['locale'],'Device
OS':_col4['device.os'])), _col4[_col8]) (type: string)
outputColumnNames: _col0, _col1, _col2, _col3, _col4,
_col5, _col7, _col8
Statistics: Num rows: 144917644 Data size: 147816001962
Basic stats: COMPLETE Column stats: NONE
Lateral View Forward
Statistics: Num rows: 144917644 Data size:
147816001962 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: _col0 (type: string), _col1 (type:
string), _col2 (type: string), _col3 (type: string), _col4 (type: double),
_col5 (type: bigint), _col7 (type: string)
outputColumnNames: _col0, _col1, _col2, _col3,
_col4, _col5, _col7
Statistics: Num rows: 144917644 Data size:
147816001962 Basic stats: COMPLETE Column stats: NONE
Lateral View Join Operator
outputColumnNames: _col0, _col1, _col2, _col3,
_col4, _col5, _col7, _col9, _col10, _col11, _col12, _col13, _col14, _col15
Statistics: Num rows: 289835288 Data size:
295632003924 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: _col0 (type: string), _col1 (type:
string), _col2 (type: string), _col3 (type: string), _col4 (type: double),
_col5 (type: bigint), _col7 (type: string), _col9 (type: map<string,string>)
outputColumnNames: _col0, _col1, _col2, _col3,
_col4, _col5, _col7, _col8
Statistics: Num rows: 289835288 Data size:
295632003924 Basic stats: COMPLETE Column stats: NONE
Lateral View Forward
Statistics: Num rows: 289835288 Data size:
295632003924 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: _col0 (type: string), _col1
(type: string), _col2 (type: string), _col3 (type: string), _col4 (type:
double), _col5 (type: bigint), _col7 (type: string)
outputColumnNames: _col0, _col1, _col2,
_col3, _col4, _col5, _col7
Statistics: Num rows: 289835288 Data size:
295632003924 Basic stats: COMPLETE Column stats: NONE
Lateral View Join Operator
outputColumnNames: _col0, _col1, _col2,
_col3, _col4, _col5, _col7, _col9, _col10
Statistics: Num rows: 579670576 Data
size: 591264007848 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: _col0 (type: string),
_col1 (type: string), _col2 (type: string), _col3 (type: string), _col4 (type:
double), _col5 (type: bigint), _col7 (type: string), _col9 (type: string),
_col10 (type: string)
outputColumnNames: _col0, _col1, _col2,
_col3, _col4, _col5, _col7, _col8, _col9
Statistics: Num rows: 579670576 Data
size: 591264007848 Basic stats: COMPLETE Column stats: NONE
File Output Operator
compressed: true
table:
input format:
org.apache.hadoop.mapred.SequenceFileInputFormat
output format:
org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
serde:
org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
Select Operator
expressions: _col8 (type:
map<string,string>)
outputColumnNames: _col0
Statistics: Num rows: 289835288 Data size:
295632003924 Basic stats: COMPLETE Column stats: NONE
UDTF Operator
Statistics: Num rows: 289835288 Data
size: 295632003924 Basic stats: COMPLETE Column stats: NONE
function name: explode
Lateral View Join Operator
outputColumnNames: _col0, _col1, _col2,
_col3, _col4, _col5, _col7, _col9, _col10
Statistics: Num rows: 579670576 Data
size: 591264007848 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: _col0 (type: string),
_col1 (type: string), _col2 (type: string), _col3 (type: string), _col4 (type:
double), _col5 (type: bigint), _col7 (type: string), _col9 (type: string),
_col10 (type: string)
outputColumnNames: _col0, _col1,
_col2, _col3, _col4, _col5, _col7, _col8, _col9
Statistics: Num rows: 579670576 Data
size: 591264007848 Basic stats: COMPLETE Column stats: NONE
File Output Operator
compressed: true
table:
input format:
org.apache.hadoop.mapred.SequenceFileInputFormat
output format:
org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
serde:
org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
Select Operator
expressions: 'map<string,string>' (type: string),
'pub_data' (type: string), map('pub_data':_col8) (type: map<string,string>),
'0000000000' (type: string)
outputColumnNames: _col0, _col1, _col2, _col3
Statistics: Num rows: 144917644 Data size:
147816001962 Basic stats: COMPLETE Column stats: NONE
UDTF Operator
Statistics: Num rows: 144917644 Data size:
147816001962 Basic stats: COMPLETE Column stats: NONE
function name: up_explode_map
Lateral View Join Operator
outputColumnNames: _col0, _col1, _col2, _col3,
_col4, _col5, _col7, _col9, _col10, _col11, _col12, _col13, _col14, _col15
Statistics: Num rows: 289835288 Data size:
295632003924 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: _col0 (type: string), _col1
(type: string), _col2 (type: string), _col3 (type: string), _col4 (type:
double), _col5 (type: bigint), _col7 (type: string), _col9 (type:
map<string,string>)
outputColumnNames: _col0, _col1, _col2,
_col3, _col4, _col5, _col7, _col8
Statistics: Num rows: 289835288 Data size:
295632003924 Basic stats: COMPLETE Column stats: NONE
Lateral View Forward
Statistics: Num rows: 289835288 Data size:
295632003924 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: _col0 (type: string), _col1
(type: string), _col2 (type: string), _col3 (type: string), _col4 (type:
double), _col5 (type: bigint), _col7 (type: string)
outputColumnNames: _col0, _col1, _col2,
_col3, _col4, _col5, _col7
Statistics: Num rows: 289835288 Data
size: 295632003924 Basic stats: COMPLETE Column stats: NONE
Lateral View Join Operator
outputColumnNames: _col0, _col1, _col2,
_col3, _col4, _col5, _col7, _col9, _col10
Statistics: Num rows: 579670576 Data
size: 591264007848 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: _col0 (type: string),
_col1 (type: string), _col2 (type: string), _col3 (type: string), _col4 (type:
double), _col5 (type: bigint), _col7 (type: string), _col9 (type: string),
_col10 (type: string)
outputColumnNames: _col0, _col1,
_col2, _col3, _col4, _col5, _col7, _col8, _col9
Statistics: Num rows: 579670576 Data
size: 591264007848 Basic stats: COMPLETE Column stats: NONE
File Output Operator
compressed: true
table:
input format:
org.apache.hadoop.mapred.SequenceFileInputFormat
output format:
org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
serde:
org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
Select Operator
expressions: _col8 (type:
map<string,string>)
outputColumnNames: _col0
Statistics: Num rows: 289835288 Data
size: 295632003924 Basic stats: COMPLETE Column stats: NONE
UDTF Operator
Statistics: Num rows: 289835288 Data
size: 295632003924 Basic stats: COMPLETE Column stats: NONE
function name: explode
Lateral View Join Operator
outputColumnNames: _col0, _col1,
_col2, _col3, _col4, _col5, _col7, _col9, _col10
Statistics: Num rows: 579670576 Data
size: 591264007848 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: _col0 (type: string),
_col1 (type: string), _col2 (type: string), _col3 (type: string), _col4 (type:
double), _col5 (type: bigint), _col7 (type: string), _col9 (type: string),
_col10 (type: string)
outputColumnNames: _col0, _col1,
_col2, _col3, _col4, _col5, _col7, _col8, _col9
Statistics: Num rows: 579670576
Data size: 591264007848 Basic stats: COMPLETE Column stats: NONE
File Output Operator
compressed: true
table:
input format:
org.apache.hadoop.mapred.SequenceFileInputFormat
output format:
org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
serde:
org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
Select Operator
expressions: array('pub_data','user_attributes','standard')
(type: array<string>)
outputColumnNames: _col0
Statistics: Num rows: 72458822 Data size: 73908000981 Basic
stats: COMPLETE Column stats: NONE
UDTF Operator
Statistics: Num rows: 72458822 Data size: 73908000981
Basic stats: COMPLETE Column stats: NONE
function name: explode
Lateral View Join Operator
outputColumnNames: _col0, _col1, _col2, _col3, _col5,
_col6, _col4, _col8
Statistics: Num rows: 144917644 Data size: 147816001962
Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: _col0 (type: string), _col1 (type:
string), _col2 (type: string), _col3 (type: string), _col5 (type: double),
_col6 (type: bigint), _col8 (type: string), if((_col8 = 'standard'),
to_json(map('Country':_col4['location.country_ip'],'App
Version':_col4['app.version'],'OS Version':_col4['device.os_version'],'Device
Type':_col4['device.type'],'Device
Manufacturer':_col4['device.manufacturer'],'Connection
Type':_col4['device.connection'],'Device
Carrier':_col4['device.carrier'],'Language':_col4['locale'],'Device
OS':_col4['device.os'])), _col4[_col8]) (type: string)
outputColumnNames: _col0, _col1, _col2, _col3, _col4,
_col5, _col7, _col8
Statistics: Num rows: 144917644 Data size:
147816001962 Basic stats: COMPLETE Column stats: NONE
Lateral View Forward
Statistics: Num rows: 144917644 Data size:
147816001962 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: _col0 (type: string), _col1 (type:
string), _col2 (type: string), _col3 (type: string), _col4 (type: double),
_col5 (type: bigint), _col7 (type: string)
outputColumnNames: _col0, _col1, _col2, _col3,
_col4, _col5, _col7
Statistics: Num rows: 144917644 Data size:
147816001962 Basic stats: COMPLETE Column stats: NONE
Lateral View Join Operator
outputColumnNames: _col0, _col1, _col2, _col3,
_col4, _col5, _col7, _col9, _col10, _col11, _col12, _col13, _col14, _col15
Statistics: Num rows: 289835288 Data size:
295632003924 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: _col0 (type: string), _col1
(type: string), _col2 (type: string), _col3 (type: string), _col4 (type:
double), _col5 (type: bigint), _col7 (type: string), _col9 (type:
map<string,string>)
outputColumnNames: _col0, _col1, _col2,
_col3, _col4, _col5, _col7, _col8
Statistics: Num rows: 289835288 Data size:
295632003924 Basic stats: COMPLETE Column stats: NONE
Lateral View Forward
Statistics: Num rows: 289835288 Data size:
295632003924 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: _col0 (type: string), _col1
(type: string), _col2 (type: string), _col3 (type: string), _col4 (type:
double), _col5 (type: bigint), _col7 (type: string)
outputColumnNames: _col0, _col1, _col2,
_col3, _col4, _col5, _col7
Statistics: Num rows: 289835288 Data
size: 295632003924 Basic stats: COMPLETE Column stats: NONE
Lateral View Join Operator
outputColumnNames: _col0, _col1, _col2,
_col3, _col4, _col5, _col7, _col9, _col10
Statistics: Num rows: 579670576 Data
size: 591264007848 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: _col0 (type: string),
_col1 (type: string), _col2 (type: string), _col3 (type: string), _col4 (type:
double), _col5 (type: bigint), _col7 (type: string), _col9 (type: string),
_col10 (type: string)
outputColumnNames: _col0, _col1,
_col2, _col3, _col4, _col5, _col7, _col8, _col9
Statistics: Num rows: 579670576 Data
size: 591264007848 Basic stats: COMPLETE Column stats: NONE
File Output Operator
compressed: true
table:
input format:
org.apache.hadoop.mapred.SequenceFileInputFormat
output format:
org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
serde:
org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
Select Operator
expressions: _col8 (type:
map<string,string>)
outputColumnNames: _col0
Statistics: Num rows: 289835288 Data
size: 295632003924 Basic stats: COMPLETE Column stats: NONE
UDTF Operator
Statistics: Num rows: 289835288 Data
size: 295632003924 Basic stats: COMPLETE Column stats: NONE
function name: explode
Lateral View Join Operator
outputColumnNames: _col0, _col1,
_col2, _col3, _col4, _col5, _col7, _col9, _col10
Statistics: Num rows: 579670576 Data
size: 591264007848 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: _col0 (type: string),
_col1 (type: string), _col2 (type: string), _col3 (type: string), _col4 (type:
double), _col5 (type: bigint), _col7 (type: string), _col9 (type: string),
_col10 (type: string)
outputColumnNames: _col0, _col1,
_col2, _col3, _col4, _col5, _col7, _col8, _col9
Statistics: Num rows: 579670576
Data size: 591264007848 Basic stats: COMPLETE Column stats: NONE
File Output Operator
compressed: true
table:
input format:
org.apache.hadoop.mapred.SequenceFileInputFormat
output format:
org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
serde:
org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
Select Operator
expressions: 'map<string,string>' (type: string),
'pub_data' (type: string), map('pub_data':_col8) (type: map<string,string>),
'0000000000' (type: string)
outputColumnNames: _col0, _col1, _col2, _col3
Statistics: Num rows: 144917644 Data size:
147816001962 Basic stats: COMPLETE Column stats: NONE
UDTF Operator
Statistics: Num rows: 144917644 Data size:
147816001962 Basic stats: COMPLETE Column stats: NONE
function name: up_explode_map
Lateral View Join Operator
outputColumnNames: _col0, _col1, _col2,
_col3, _col4, _col5, _col7, _col9, _col10, _col11, _col12, _col13, _col14,
_col15
Statistics: Num rows: 289835288 Data size:
295632003924 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: _col0 (type: string), _col1
(type: string), _col2 (type: string), _col3 (type: string), _col4 (type:
double), _col5 (type: bigint), _col7 (type: string), _col9 (type:
map<string,string>)
outputColumnNames: _col0, _col1, _col2,
_col3, _col4, _col5, _col7, _col8
Statistics: Num rows: 289835288 Data size:
295632003924 Basic stats: COMPLETE Column stats: NONE
Lateral View Forward
Statistics: Num rows: 289835288 Data
size: 295632003924 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: _col0 (type: string),
_col1 (type: string), _col2 (type: string), _col3 (type: string), _col4 (type:
double), _col5 (type: bigint), _col7 (type: string)
outputColumnNames: _col0, _col1, _col2,
_col3, _col4, _col5, _col7
Statistics: Num rows: 289835288 Data
size: 295632003924 Basic stats: COMPLETE Column stats: NONE
Lateral View Join Operator
outputColumnNames: _col0, _col1,
_col2, _col3, _col4, _col5, _col7, _col9, _col10
Statistics: Num rows: 579670576 Data
size: 591264007848 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: _col0 (type: string),
_col1 (type: string), _col2 (type: string), _col3 (type: string), _col4 (type:
double), _col5 (type: bigint), _col7 (type: string), _col9 (type: string),
_col10 (type: string)
outputColumnNames: _col0, _col1,
_col2, _col3, _col4, _col5, _col7, _col8, _col9
Statistics: Num rows: 579670576
Data size: 591264007848 Basic stats: COMPLETE Column stats: NONE
File Output Operator
compressed: true
table:
input format:
org.apache.hadoop.mapred.SequenceFileInputFormat
output format:
org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
serde:
org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
Select Operator
expressions: _col8 (type:
map<string,string>)
outputColumnNames: _col0
Statistics: Num rows: 289835288 Data
size: 295632003924 Basic stats: COMPLETE Column stats: NONE
UDTF Operator
Statistics: Num rows: 289835288 Data
size: 295632003924 Basic stats: COMPLETE Column stats: NONE
function name: explode
Lateral View Join Operator
outputColumnNames: _col0, _col1,
_col2, _col3, _col4, _col5, _col7, _col9, _col10
Statistics: Num rows: 579670576
Data size: 591264007848 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: _col0 (type:
string), _col1 (type: string), _col2 (type: string), _col3 (type: string),
_col4 (type: double), _col5 (type: bigint), _col7 (type: string), _col9 (type:
string), _col10 (type: string)
outputColumnNames: _col0, _col1,
_col2, _col3, _col4, _col5, _col7, _col8, _col9
Statistics: Num rows: 579670576
Data size: 591264007848 Basic stats: COMPLETE Column stats: NONE
File Output Operator
compressed: true
table:
input format:
org.apache.hadoop.mapred.SequenceFileInputFormat
output format:
org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
serde:
org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
Stage: Stage-6
Map Reduce
Map Operator Tree:
TableScan
Reduce Output Operator
key expressions: _col0 (type: string), _col1 (type: string),
_col7 (type: string), _col8 (type: string), _col9 (type: string), _col4 (type:
double), _col0 (type: string), _col1 (type: string), _col7 (type: string),
_col8 (type: string), _col9 (type: string), _col4 (type: double)
sort order: ++++++++++++
Map-reduce partition columns: _col0 (type: string), _col1 (type:
string), _col7 (type: string), _col8 (type: string), _col9 (type: string),
_col4 (type: double)
Statistics: Num rows: 579670576 Data size: 591264007848 Basic
stats: COMPLETE Column stats: NONE
value expressions: _col2 (type: string), _col3 (type: string),
_col5 (type: bigint)
Reduce Operator Tree:
Select Operator
expressions: KEY.reducesinkkey0 (type: string), KEY.reducesinkkey1
(type: string), VALUE._col0 (type: string), VALUE._col1 (type: string),
KEY.reducesinkkey5 (type: double), VALUE._col2 (type: bigint),
KEY.reducesinkkey2 (type: string), KEY.reducesinkkey3 (type: string),
KEY.reducesinkkey4 (type: string)
outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col7,
_col8, _col9
Statistics: Num rows: 579670576 Data size: 591264007848 Basic stats:
COMPLETE Column stats: NONE
PTF Operator
Statistics: Num rows: 579670576 Data size: 591264007848 Basic
stats: COMPLETE Column stats: NONE
File Output Operator
compressed: true
table:
input format: org.apache.hadoop.mapred.SequenceFileInputFormat
output format:
org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
serde:
org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
Stage: Stage-7
Map Reduce
Map Operator Tree:
TableScan
Reduce Output Operator
key expressions: _col0 (type: string), _col1 (type: string),
_col7 (type: string), _col8 (type: string), _col4 (type: double), _col0 (type:
string), _col1 (type: string), _col7 (type: string), _col8 (type: string),
_col4 (type: double)
sort order: ++++++++++
Map-reduce partition columns: _col0 (type: string), _col1 (type:
string), _col7 (type: string), _col8 (type: string), _col4 (type: double)
Statistics: Num rows: 579670576 Data size: 591264007848 Basic
stats: COMPLETE Column stats: NONE
value expressions: _wcol0 (type: bigint), _wcol1 (type: int),
_col2 (type: string), _col3 (type: string), _col5 (type: bigint), _col9 (type:
string)
Reduce Operator Tree:
Select Operator
expressions: VALUE._col0 (type: bigint), VALUE._col1 (type: int),
KEY.reducesinkkey0 (type: string), KEY.reducesinkkey1 (type: string),
VALUE._col2 (type: string), VALUE._col3 (type: string), KEY.reducesinkkey4
(type: double), VALUE._col4 (type: bigint), KEY.reducesinkkey2 (type: string),
KEY.reducesinkkey3 (type: string), VALUE._col6 (type: string)
outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6,
_col7, _col9, _col10, _col11
Statistics: Num rows: 579670576 Data size: 591264007848 Basic stats:
COMPLETE Column stats: NONE
PTF Operator
Statistics: Num rows: 579670576 Data size: 591264007848 Basic
stats: COMPLETE Column stats: NONE
Select Operator
expressions: _col0 (type: bigint), _col1 (type: int), _col9
(type: string), _col10 (type: string), _col11 (type: string), _col6 (type:
double), _wcol2 (type: bigint), _wcol3 (type: int), _col2 (type: string), _col3
(type: string), _col4 (type: string), _col5 (type: string), _col7 (type: bigint)
outputColumnNames: _col0, _col1, _col10, _col11, _col12, _col13,
_col2, _col3, _col4, _col5, _col6, _col7, _col8
Statistics: Num rows: 579670576 Data size: 591264007848 Basic
stats: COMPLETE Column stats: NONE
File Output Operator
compressed: true
table:
input format:
org.apache.hadoop.mapred.SequenceFileInputFormat
output format:
org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
serde:
org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
Stage: Stage-8
Map Reduce
Map Operator Tree:
TableScan
Reduce Output Operator
key expressions: _col4 (type: string), _col5 (type: string),
_col10 (type: string), _col11 (type: string), _col12 (type: string), _col1
(type: int), _col4 (type: string), _col5 (type: string), _col10 (type: string),
_col11 (type: string), _col12 (type: string), _col1 (type: int)
sort order: ++++++++++++
Map-reduce partition columns: _col4 (type: string), _col5 (type:
string), _col10 (type: string), _col11 (type: string), _col12 (type: string),
_col1 (type: int)
Statistics: Num rows: 579670576 Data size: 591264007848 Basic
stats: COMPLETE Column stats: NONE
value expressions: _col0 (type: bigint), _col2 (type: bigint),
_col3 (type: int), _col6 (type: string), _col7 (type: string), _col8 (type:
bigint), _col13 (type: double)
Reduce Operator Tree:
Select Operator
expressions: VALUE._col0 (type: bigint), KEY.reducesinkkey5 (type:
int), VALUE._col1 (type: bigint), VALUE._col2 (type: int), KEY.reducesinkkey0
(type: string), KEY.reducesinkkey1 (type: string), VALUE._col3 (type: string),
VALUE._col4 (type: string), VALUE._col5 (type: bigint), KEY.reducesinkkey2
(type: string), KEY.reducesinkkey3 (type: string), KEY.reducesinkkey4 (type:
string), VALUE._col7 (type: double)
outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6,
_col7, _col8, _col10, _col11, _col12, _col13
Statistics: Num rows: 579670576 Data size: 591264007848 Basic stats:
COMPLETE Column stats: NONE
PTF Operator
Statistics: Num rows: 579670576 Data size: 591264007848 Basic
stats: COMPLETE Column stats: NONE
File Output Operator
compressed: true
table:
input format: org.apache.hadoop.mapred.SequenceFileInputFormat
output format:
org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
serde:
org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
Stage: Stage-9
Map Reduce
Map Operator Tree:
TableScan
Reduce Output Operator
key expressions: _col4 (type: string), _col5 (type: string),
_col10 (type: string), _col11 (type: string), _col3 (type: int), _col4 (type:
string), _col5 (type: string), _col10 (type: string), _col11 (type: string),
_col3 (type: int)
sort order: ++++++++++
Map-reduce partition columns: _col4 (type: string), _col5 (type:
string), _col10 (type: string), _col11 (type: string), _col3 (type: int)
Statistics: Num rows: 579670576 Data size: 591264007848 Basic
stats: COMPLETE Column stats: NONE
value expressions: _wcol0 (type: bigint), _col1 (type: int),
_col2 (type: bigint), _col6 (type: string), _col7 (type: string), _col8 (type:
bigint), _col12 (type: string), _col13 (type: double)
Reduce Operator Tree:
Select Operator
expressions: VALUE._col0 (type: bigint), VALUE._col2 (type: int),
VALUE._col3 (type: bigint), KEY.reducesinkkey4 (type: int), KEY.reducesinkkey0
(type: string), KEY.reducesinkkey1 (type: string), VALUE._col4 (type: string),
VALUE._col5 (type: string), VALUE._col6 (type: bigint), KEY.reducesinkkey2
(type: string), KEY.reducesinkkey3 (type: string), VALUE._col8 (type: string),
VALUE._col9 (type: double)
outputColumnNames: _col0, _col2, _col3, _col4, _col5, _col6, _col7,
_col8, _col9, _col11, _col12, _col13, _col14
Statistics: Num rows: 579670576 Data size: 591264007848 Basic stats:
COMPLETE Column stats: NONE
PTF Operator
Statistics: Num rows: 579670576 Data size: 591264007848 Basic
stats: COMPLETE Column stats: NONE
Select Operator
expressions: _col0 (type: bigint), _wcol1 (type: bigint), _col13
(type: string), _col14 (type: double), _col2 (type: int), _col4 (type: int),
_col5 (type: string), _col6 (type: string), _col7 (type: string), _col8 (type:
string), _col9 (type: bigint), _col11 (type: string), _col12 (type: string)
outputColumnNames: _col0, _col1, _col10, _col11, _col12, _col13,
_col2, _col3, _col4, _col5, _col6, _col8, _col9
Statistics: Num rows: 579670576 Data size: 591264007848 Basic
stats: COMPLETE Column stats: NONE
File Output Operator
compressed: true
table:
input format:
org.apache.hadoop.mapred.SequenceFileInputFormat
output format:
org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
serde:
org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
Stage: Stage-10
Map Reduce
Map Operator Tree:
TableScan
Reduce Output Operator
key expressions: _col11 (type: double), _col2 (type: string),
_col3 (type: string), _col8 (type: string), _col9 (type: string), _col10 (type:
string), _col11 (type: double), _col2 (type: string), _col3 (type: string),
_col8 (type: string), _col9 (type: string), _col10 (type: string)
sort order: ++++++++++++
Map-reduce partition columns: _col11 (type: double), _col2 (type:
string), _col3 (type: string), _col8 (type: string), _col9 (type: string),
_col10 (type: string)
Statistics: Num rows: 579670576 Data size: 591264007848 Basic
stats: COMPLETE Column stats: NONE
value expressions: _col0 (type: bigint), _col1 (type: bigint),
_col4 (type: string), _col5 (type: string), _col6 (type: bigint), _col12 (type:
int), _col13 (type: int)
Reduce Operator Tree:
Select Operator
expressions: VALUE._col0 (type: bigint), VALUE._col1 (type: bigint),
KEY.reducesinkkey1 (type: string), KEY.reducesinkkey2 (type: string),
VALUE._col2 (type: string), VALUE._col3 (type: string), VALUE._col4 (type:
bigint), KEY.reducesinkkey3 (type: string), KEY.reducesinkkey4 (type: string),
KEY.reducesinkkey5 (type: string), KEY.reducesinkkey0 (type: double),
VALUE._col6 (type: int), VALUE._col7 (type: int)
outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6,
_col8, _col9, _col10, _col11, _col12, _col13
Statistics: Num rows: 579670576 Data size: 591264007848 Basic stats:
COMPLETE Column stats: NONE
PTF Operator
Statistics: Num rows: 579670576 Data size: 591264007848 Basic
stats: COMPLETE Column stats: NONE
File Output Operator
compressed: true
table:
input format: org.apache.hadoop.mapred.SequenceFileInputFormat
output format:
org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
serde:
org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
Stage: Stage-11
Map Reduce
Map Operator Tree:
TableScan
Reduce Output Operator
key expressions: _col11 (type: double), _col2 (type: string),
_col3 (type: string), _col8 (type: string), _col9 (type: string), _col11 (type:
double), _col2 (type: string), _col3 (type: string), _col8 (type: string),
_col9 (type: string)
sort order: ++++++++++
Map-reduce partition columns: _col11 (type: double), _col2 (type:
string), _col3 (type: string), _col8 (type: string), _col9 (type: string)
Statistics: Num rows: 579670576 Data size: 591264007848 Basic
stats: COMPLETE Column stats: NONE
value expressions: _wcol0 (type: bigint), _col1 (type: bigint),
_col4 (type: string), _col5 (type: string), _col6 (type: bigint), _col10 (type:
string), _col12 (type: int), _col13 (type: int)
Reduce Operator Tree:
Select Operator
expressions: VALUE._col0 (type: bigint), VALUE._col2 (type: bigint),
KEY.reducesinkkey1 (type: string), KEY.reducesinkkey2 (type: string),
VALUE._col3 (type: string), VALUE._col4 (type: string), VALUE._col5 (type:
bigint), KEY.reducesinkkey3 (type: string), KEY.reducesinkkey4 (type: string),
VALUE._col7 (type: string), KEY.reducesinkkey0 (type: double), VALUE._col8
(type: int), VALUE._col9 (type: int)
outputColumnNames: _col0, _col2, _col3, _col4, _col5, _col6, _col7,
_col9, _col10, _col11, _col12, _col13, _col14
Statistics: Num rows: 579670576 Data size: 591264007848 Basic stats:
COMPLETE Column stats: NONE
PTF Operator
Statistics: Num rows: 579670576 Data size: 591264007848 Basic
stats: COMPLETE Column stats: NONE
Select Operator
expressions: _col0 (type: bigint), _wcol1 (type: bigint), _col11
(type: string), _col13 (type: int), _col14 (type: int), _col12 (type: double),
_col3 (type: string), _col4 (type: string), _col5 (type: string), _col6 (type:
string), _col7 (type: bigint), _col9 (type: string), _col10 (type: string)
outputColumnNames: _col0, _col1, _col10, _col11, _col12, _col13,
_col2, _col3, _col4, _col5, _col6, _col8, _col9
Statistics: Num rows: 579670576 Data size: 591264007848 Basic
stats: COMPLETE Column stats: NONE
File Output Operator
compressed: true
table:
input format:
org.apache.hadoop.mapred.SequenceFileInputFormat
output format:
org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
serde:
org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
Stage: Stage-12
Map Reduce
Map Operator Tree:
TableScan
Reduce Output Operator
key expressions: _col12 (type: int), _col2 (type: string), _col3
(type: string), _col8 (type: string), _col1 (type: bigint), _col9 (type: string)
sort order: ++++-+
Map-reduce partition columns: _col12 (type: int), _col2 (type:
string), _col3 (type: string), _col8 (type: string)
Statistics: Num rows: 579670576 Data size: 591264007848 Basic
stats: COMPLETE Column stats: NONE
value expressions: _col0 (type: bigint), _col4 (type: string),
_col5 (type: string), _col6 (type: bigint), _col10 (type: string), _col11
(type: int), _col13 (type: double)
Reduce Operator Tree:
Select Operator
expressions: VALUE._col0 (type: bigint), KEY.reducesinkkey4 (type:
bigint), KEY.reducesinkkey1 (type: string), KEY.reducesinkkey2 (type: string),
VALUE._col1 (type: string), VALUE._col2 (type: string), VALUE._col3 (type:
bigint), KEY.reducesinkkey3 (type: string), KEY.reducesinkkey5 (type: string),
VALUE._col5 (type: string), VALUE._col6 (type: int), KEY.reducesinkkey0 (type:
int), VALUE._col7 (type: double)
outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6,
_col8, _col9, _col10, _col11, _col12, _col13
Statistics: Num rows: 579670576 Data size: 591264007848 Basic stats:
COMPLETE Column stats: NONE
PTF Operator
Statistics: Num rows: 579670576 Data size: 591264007848 Basic
stats: COMPLETE Column stats: NONE
File Output Operator
compressed: true
table:
input format: org.apache.hadoop.mapred.SequenceFileInputFormat
output format:
org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
serde:
org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
Stage: Stage-13
Map Reduce
Map Operator Tree:
TableScan
Reduce Output Operator
key expressions: _col11 (type: int), _col2 (type: string), _col3
(type: string), _col8 (type: string), _col9 (type: string), _col0 (type:
bigint), _col10 (type: string)
sort order: +++++-+
Map-reduce partition columns: _col11 (type: int), _col2 (type:
string), _col3 (type: string), _col8 (type: string), _col9 (type: string)
Statistics: Num rows: 579670576 Data size: 591264007848 Basic
stats: COMPLETE Column stats: NONE
value expressions: _wcol0 (type: int), _col4 (type: string),
_col5 (type: string), _col6 (type: bigint), _col12 (type: int), _col13 (type:
double)
Reduce Operator Tree:
Select Operator
expressions: VALUE._col0 (type: int), KEY.reducesinkkey5 (type:
bigint), KEY.reducesinkkey1 (type: string), KEY.reducesinkkey2 (type: string),
VALUE._col2 (type: string), VALUE._col3 (type: string), VALUE._col4 (type:
bigint), KEY.reducesinkkey3 (type: string), KEY.reducesinkkey4 (type: string),
KEY.reducesinkkey6 (type: string), KEY.reducesinkkey0 (type: int), VALUE._col6
(type: int), VALUE._col7 (type: double)
outputColumnNames: _col0, _col1, _col3, _col4, _col5, _col6, _col7,
_col9, _col10, _col11, _col12, _col13, _col14
Statistics: Num rows: 579670576 Data size: 591264007848 Basic stats:
COMPLETE Column stats: NONE
PTF Operator
Statistics: Num rows: 579670576 Data size: 591264007848 Basic
stats: COMPLETE Column stats: NONE
Select Operator
expressions: _col3 (type: string), _col4 (type: string), _col13
(type: int), _col14 (type: double), _col0 (type: int), _wcol1 (type: int),
_col5 (type: string), _col6 (type: string), _col7 (type: bigint), _col9 (type:
string), _col10 (type: string), _col11 (type: string)
outputColumnNames: _col0, _col1, _col10, _col11, _col12, _col13,
_col2, _col3, _col4, _col6, _col7, _col8
Statistics: Num rows: 579670576 Data size: 591264007848 Basic
stats: COMPLETE Column stats: NONE
File Output Operator
compressed: true
table:
input format:
org.apache.hadoop.mapred.SequenceFileInputFormat
output format:
org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
serde:
org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
Stage: Stage-14
Map Reduce
Map Operator Tree:
TableScan
Reduce Output Operator
key expressions: _col11 (type: double), _col0 (type: string),
_col1 (type: string), _col6 (type: string), _col7 (type: string), _col11 (type:
double), _col0 (type: string), _col1 (type: string), _col6 (type: string),
_col7 (type: string)
sort order: ++++++++++
Map-reduce partition columns: _col11 (type: double), _col0 (type:
string), _col1 (type: string), _col6 (type: string), _col7 (type: string)
Statistics: Num rows: 579670576 Data size: 591264007848 Basic
stats: COMPLETE Column stats: NONE
value expressions: _col2 (type: string), _col3 (type: string),
_col4 (type: bigint), _col8 (type: string), _col10 (type: int), _col12 (type:
int), _col13 (type: int)
Reduce Operator Tree:
Select Operator
expressions: KEY.reducesinkkey1 (type: string), KEY.reducesinkkey2
(type: string), VALUE._col0 (type: string), VALUE._col1 (type: string),
VALUE._col2 (type: bigint), KEY.reducesinkkey3 (type: string),
KEY.reducesinkkey4 (type: string), VALUE._col4 (type: string), VALUE._col6
(type: int), KEY.reducesinkkey0 (type: double), VALUE._col7 (type: int),
VALUE._col8 (type: int)
outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col6, _col7,
_col8, _col10, _col11, _col12, _col13
Statistics: Num rows: 579670576 Data size: 591264007848 Basic stats:
COMPLETE Column stats: NONE
PTF Operator
Statistics: Num rows: 579670576 Data size: 591264007848 Basic
stats: COMPLETE Column stats: NONE
File Output Operator
compressed: true
table:
input format: org.apache.hadoop.mapred.SequenceFileInputFormat
output format:
org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
serde:
org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
Stage: Stage-15
Map Reduce
Map Operator Tree:
TableScan
Reduce Output Operator
key expressions: _col11 (type: double), _col0 (type: string),
_col1 (type: string), _col6 (type: string), _col7 (type: string), _col8 (type:
string), _col11 (type: double), _col0 (type: string), _col1 (type: string),
_col6 (type: string), _col7 (type: string), _col8 (type: string)
sort order: ++++++++++++
Map-reduce partition columns: _col11 (type: double), _col0 (type:
string), _col1 (type: string), _col6 (type: string), _col7 (type: string),
_col8 (type: string)
Statistics: Num rows: 579670576 Data size: 591264007848 Basic
stats: COMPLETE Column stats: NONE
value expressions: _wcol0 (type: int), _col2 (type: string),
_col3 (type: string), _col4 (type: bigint), _col10 (type: int), _col13 (type:
int)
Reduce Operator Tree:
Select Operator
expressions: VALUE._col0 (type: int), KEY.reducesinkkey1 (type:
string), KEY.reducesinkkey2 (type: string), VALUE._col1 (type: string),
VALUE._col2 (type: string), VALUE._col3 (type: bigint), KEY.reducesinkkey3
(type: string), KEY.reducesinkkey4 (type: string), KEY.reducesinkkey5 (type:
string), VALUE._col6 (type: int), KEY.reducesinkkey0 (type: double),
VALUE._col8 (type: int)
outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col7,
_col8, _col9, _col11, _col12, _col14
Statistics: Num rows: 579670576 Data size: 591264007848 Basic stats:
COMPLETE Column stats: NONE
PTF Operator
Statistics: Num rows: 579670576 Data size: 591264007848 Basic
stats: COMPLETE Column stats: NONE
Filter Operator
predicate: (_col0 <= 50) (type: boolean)
Statistics: Num rows: 193223525 Data size: 197088002275 Basic
stats: COMPLETE Column stats: NONE
Select Operator
expressions: _col1 (type: string), _col2 (type: string), _col7
(type: string), _col8 (type: string), _wcol1 (type: int), _col9 (type: string),
_col3 (type: string), _col4 (type: string), _col5 (type: bigint)
outputColumnNames: _col0, _col1, _col6, _col7, _col12, _col8,
_col2, _col3, _col4
Statistics: Num rows: 193223525 Data size: 197088002275 Basic
stats: COMPLETE Column stats: NONE
Group By Operator
aggregations: count(1), count(DISTINCT _col2), count(DISTINCT
_col2, _col3), max(_col4)
keys: _col0 (type: string), _col1 (type: string), _col6
(type: string), _col7 (type: string), if((_col12 <= 50), _col8, 'Others')
(type: string), _col2 (type: string), _col3 (type: string)
mode: hash
outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5,
_col6, _col7, _col8, _col9, _col10
Statistics: Num rows: 193223525 Data size: 197088002275 Basic
stats: COMPLETE Column stats: NONE
File Output Operator
compressed: true
table:
input format:
org.apache.hadoop.mapred.SequenceFileInputFormat
output format:
org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
serde:
org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
Stage: Stage-16
Map Reduce
Map Operator Tree:
TableScan
Reduce Output Operator
key expressions: _col0 (type: string), _col1 (type: string),
_col2 (type: string), _col3 (type: string), _col4 (type: string), _col5 (type:
string), _col6 (type: string)
sort order: +++++++
Map-reduce partition columns: _col0 (type: string), _col1 (type:
string), _col2 (type: string), _col3 (type: string), _col4 (type: string)
Statistics: Num rows: 193223525 Data size: 197088002275 Basic
stats: COMPLETE Column stats: NONE
value expressions: _col7 (type: bigint), _col10 (type: bigint)
Reduce Operator Tree:
Group By Operator
aggregations: count(VALUE._col0), count(DISTINCT KEY._col5:0._col0),
count(DISTINCT KEY._col5:1._col0, KEY._col5:1._col1), max(VALUE._col3)
keys: KEY._col0 (type: string), KEY._col1 (type: string), KEY._col2
(type: string), KEY._col3 (type: string), KEY._col4 (type: string)
mode: mergepartial
outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6,
_col7, _col8
Statistics: Num rows: 96611762 Data size: 98544000627 Basic stats:
COMPLETE Column stats: NONE
Select Operator
expressions: _col0 (type: string), '2016-02-03T00:00:00Z' (type:
string), _col1 (type: string), _col2 (type: string), _col3 (type: string),
_col4 (type: string), _col5 (type: bigint), _col6 (type: bigint), _col7 (type:
bigint), _col8 (type: bigint)
outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6,
_col7, _col8, _col9
Statistics: Num rows: 96611762 Data size: 98544000627 Basic stats:
COMPLETE Column stats: NONE
File Output Operator
compressed: false
Statistics: Num rows: 96611762 Data size: 98544000627 Basic
stats: COMPLETE Column stats: NONE
table:
input format: org.apache.hadoop.mapred.TextInputFormat
output format:
org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
Stage: Stage-0
Fetch Operator
limit: -1
Processor Tree:
ListSink
