Whoops.. meant to send the tez explain earlier. Here's the Tez query plan.
Good to know there's a fix .. Is there a jira that talks about this issue? Coz
I couldn't find one. Maybe I can alter the query a bit to filter these out.
Cheers,
-Gautam.
On Wed, Feb 10, 2016 at 7:32 PM, Gopal Vijayaraghavan <gop...@apache.org>
wrote:
> Hey,
>
> > Trying to benchmark with Hive on Tez causes the following error.
> >Admittedly these are some very large looking records .. the same job runs
> >fine on MR2.
> ...
> > I'v attached the query explain tree. It fails in the very last reducer
> >phase ..
>
> Can you attach the explain plan with hive.execution.engine=tez (even
> better would be the JSON output of "explain formatted").
>
> > Error: Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> >org.apache.tez.runtime.library.common.sort.impl.ExternalSorter$MapBufferTo
> >oSmallException: Record too large for in-memory buffer. Exceeded buffer
> >overflow limit, bufferOverflowRecursion=2, bufferList.size=1,
> >blockSize=268435456
>
> You have a single row which is > 200Mb?
>
> The tez 0.8.2 out there should already have this issue fixed, but single
> spill record mode is likely to spin your disks all wrong (unless you have
> SSDs).
>
> I'm guessing there's a UDTF + PTF producing humongous rows & that it's
> being planned wrong because that's not a standard pattern.
>
> Cheers,
> Gopal
>
>
>
--
"If you really want something in this life, you have to work for it. Now,
quiet! They're about to announce the lottery numbers..."
STAGE DEPENDENCIES:
Stage-1 is a root stage
Stage-0 depends on stages: Stage-1
STAGE PLANS:
Stage: Stage-1
Tez
Edges:
Reducer 10 <- Reducer 9 (SIMPLE_EDGE)
Reducer 11 <- Reducer 10 (SIMPLE_EDGE)
Reducer 12 <- Reducer 11 (SIMPLE_EDGE)
Reducer 13 <- Reducer 12 (SIMPLE_EDGE)
Reducer 14 <- Reducer 13 (SIMPLE_EDGE)
Reducer 15 <- Reducer 14 (SIMPLE_EDGE)
Reducer 16 <- Reducer 15 (SIMPLE_EDGE)
Reducer 17 <- Reducer 16 (SIMPLE_EDGE)
Reducer 2 <- Map 1 (SIMPLE_EDGE)
Reducer 3 <- Reducer 2 (SIMPLE_EDGE)
Reducer 4 <- Reducer 3 (SIMPLE_EDGE)
Reducer 5 <- Reducer 4 (SIMPLE_EDGE)
Reducer 6 <- Reducer 5 (SIMPLE_EDGE)
Reducer 7 <- Reducer 6 (SIMPLE_EDGE)
Reducer 8 <- Reducer 7 (SIMPLE_EDGE)
Reducer 9 <- Reducer 8 (SIMPLE_EDGE)
DagName: wand_20160211033737_639db8dd-47d6-450a-986f-31a538d29b20:1
Vertices:
Map 1
Map Operator Tree:
TableScan
alias: upsight_clean_data
Statistics: Num rows: 434752932 Data size: 443448005886 Basic
stats: COMPLETE Column stats: NONE
Filter Operator
predicate: (msg_type like 'pub.%') (type: boolean)
Statistics: Num rows: 217376466 Data size: 221724002943
Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
key expressions: value['app_id'] (type: string),
value['type'] (type: string), (value['ts'] % 1000) (type: double),
value['app_id'] (type: string), value['type'] (type: string), (value['ts'] %
1000) (type: double)
sort order: ++++++
Map-reduce partition columns: value['app_id'] (type:
string), value['type'] (type: string), (value['ts'] % 1000) (type: double)
Statistics: Num rows: 217376466 Data size: 221724002943
Basic stats: COMPLETE Column stats: NONE
value expressions: msg_type (type: string), value (type:
map<string,string>)
Reducer 10
Reduce Operator Tree:
Select Operator
expressions: VALUE._col0 (type: bigint), VALUE._col2 (type:
int), VALUE._col3 (type: bigint), KEY.reducesinkkey4 (type: int),
KEY.reducesinkkey0 (type: string), KEY.reducesinkkey1 (type: string),
VALUE._col4 (type: string), VALUE._col5 (type: string), VALUE._col6 (type:
bigint), KEY.reducesinkkey2 (type: string), KEY.reducesinkkey3 (type: string),
VALUE._col8 (type: string), VALUE._col9 (type: double)
outputColumnNames: _col0, _col2, _col3, _col4, _col5, _col6,
_col7, _col8, _col9, _col11, _col12, _col13, _col14
Statistics: Num rows: 579670576 Data size: 591264007848 Basic
stats: COMPLETE Column stats: NONE
PTF Operator
Statistics: Num rows: 579670576 Data size: 591264007848 Basic
stats: COMPLETE Column stats: NONE
Select Operator
expressions: _col0 (type: bigint), _wcol1 (type: bigint),
_col13 (type: string), _col14 (type: double), _col2 (type: int), _col4 (type:
int), _col5 (type: string), _col6 (type: string), _col7 (type: string), _col8
(type: string), _col9 (type: bigint), _col11 (type: string), _col12 (type:
string)
outputColumnNames: _col0, _col1, _col10, _col11, _col12,
_col13, _col2, _col3, _col4, _col5, _col6, _col8, _col9
Statistics: Num rows: 579670576 Data size: 591264007848
Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
key expressions: _col11 (type: double), _col2 (type:
string), _col3 (type: string), _col8 (type: string), _col9 (type: string),
_col10 (type: string), _col11 (type: double), _col2 (type: string), _col3
(type: string), _col8 (type: string), _col9 (type: string), _col10 (type:
string)
sort order: ++++++++++++
Map-reduce partition columns: _col11 (type: double),
_col2 (type: string), _col3 (type: string), _col8 (type: string), _col9 (type:
string), _col10 (type: string)
Statistics: Num rows: 579670576 Data size: 591264007848
Basic stats: COMPLETE Column stats: NONE
value expressions: _col0 (type: bigint), _col1 (type:
bigint), _col4 (type: string), _col5 (type: string), _col6 (type: bigint),
_col12 (type: int), _col13 (type: int)
Reducer 11
Reduce Operator Tree:
Select Operator
expressions: VALUE._col0 (type: bigint), VALUE._col1 (type:
bigint), KEY.reducesinkkey1 (type: string), KEY.reducesinkkey2 (type: string),
VALUE._col2 (type: string), VALUE._col3 (type: string), VALUE._col4 (type:
bigint), KEY.reducesinkkey3 (type: string), KEY.reducesinkkey4 (type: string),
KEY.reducesinkkey5 (type: string), KEY.reducesinkkey0 (type: double),
VALUE._col6 (type: int), VALUE._col7 (type: int)
outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5,
_col6, _col8, _col9, _col10, _col11, _col12, _col13
Statistics: Num rows: 579670576 Data size: 591264007848 Basic
stats: COMPLETE Column stats: NONE
PTF Operator
Statistics: Num rows: 579670576 Data size: 591264007848 Basic
stats: COMPLETE Column stats: NONE
Reduce Output Operator
key expressions: _col11 (type: double), _col2 (type:
string), _col3 (type: string), _col8 (type: string), _col9 (type: string),
_col11 (type: double), _col2 (type: string), _col3 (type: string), _col8 (type:
string), _col9 (type: string)
sort order: ++++++++++
Map-reduce partition columns: _col11 (type: double), _col2
(type: string), _col3 (type: string), _col8 (type: string), _col9 (type: string)
Statistics: Num rows: 579670576 Data size: 591264007848
Basic stats: COMPLETE Column stats: NONE
value expressions: _wcol0 (type: bigint), _col1 (type:
bigint), _col4 (type: string), _col5 (type: string), _col6 (type: bigint),
_col10 (type: string), _col12 (type: int), _col13 (type: int)
Reducer 12
Reduce Operator Tree:
Select Operator
expressions: VALUE._col0 (type: bigint), VALUE._col2 (type:
bigint), KEY.reducesinkkey1 (type: string), KEY.reducesinkkey2 (type: string),
VALUE._col3 (type: string), VALUE._col4 (type: string), VALUE._col5 (type:
bigint), KEY.reducesinkkey3 (type: string), KEY.reducesinkkey4 (type: string),
VALUE._col7 (type: string), KEY.reducesinkkey0 (type: double), VALUE._col8
(type: int), VALUE._col9 (type: int)
outputColumnNames: _col0, _col2, _col3, _col4, _col5, _col6,
_col7, _col9, _col10, _col11, _col12, _col13, _col14
Statistics: Num rows: 579670576 Data size: 591264007848 Basic
stats: COMPLETE Column stats: NONE
PTF Operator
Statistics: Num rows: 579670576 Data size: 591264007848 Basic
stats: COMPLETE Column stats: NONE
Select Operator
expressions: _col0 (type: bigint), _wcol1 (type: bigint),
_col11 (type: string), _col13 (type: int), _col14 (type: int), _col12 (type:
double), _col3 (type: string), _col4 (type: string), _col5 (type: string),
_col6 (type: string), _col7 (type: bigint), _col9 (type: string), _col10 (type:
string)
outputColumnNames: _col0, _col1, _col10, _col11, _col12,
_col13, _col2, _col3, _col4, _col5, _col6, _col8, _col9
Statistics: Num rows: 579670576 Data size: 591264007848
Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
key expressions: _col12 (type: int), _col2 (type:
string), _col3 (type: string), _col8 (type: string), _col1 (type: bigint),
_col9 (type: string)
sort order: ++++-+
Map-reduce partition columns: _col12 (type: int), _col2
(type: string), _col3 (type: string), _col8 (type: string)
Statistics: Num rows: 579670576 Data size: 591264007848
Basic stats: COMPLETE Column stats: NONE
value expressions: _col0 (type: bigint), _col4 (type:
string), _col5 (type: string), _col6 (type: bigint), _col10 (type: string),
_col11 (type: int), _col13 (type: double)
Reducer 13
Reduce Operator Tree:
Select Operator
expressions: VALUE._col0 (type: bigint), KEY.reducesinkkey4
(type: bigint), KEY.reducesinkkey1 (type: string), KEY.reducesinkkey2 (type:
string), VALUE._col1 (type: string), VALUE._col2 (type: string), VALUE._col3
(type: bigint), KEY.reducesinkkey3 (type: string), KEY.reducesinkkey5 (type:
string), VALUE._col5 (type: string), VALUE._col6 (type: int),
KEY.reducesinkkey0 (type: int), VALUE._col7 (type: double)
outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5,
_col6, _col8, _col9, _col10, _col11, _col12, _col13
Statistics: Num rows: 579670576 Data size: 591264007848 Basic
stats: COMPLETE Column stats: NONE
PTF Operator
Statistics: Num rows: 579670576 Data size: 591264007848 Basic
stats: COMPLETE Column stats: NONE
Reduce Output Operator
key expressions: _col11 (type: int), _col2 (type: string),
_col3 (type: string), _col8 (type: string), _col9 (type: string), _col0 (type:
bigint), _col10 (type: string)
sort order: +++++-+
Map-reduce partition columns: _col11 (type: int), _col2
(type: string), _col3 (type: string), _col8 (type: string), _col9 (type: string)
Statistics: Num rows: 579670576 Data size: 591264007848
Basic stats: COMPLETE Column stats: NONE
value expressions: _wcol0 (type: int), _col4 (type:
string), _col5 (type: string), _col6 (type: bigint), _col12 (type: int), _col13
(type: double)
Reducer 14
Reduce Operator Tree:
Select Operator
expressions: VALUE._col0 (type: int), KEY.reducesinkkey5 (type:
bigint), KEY.reducesinkkey1 (type: string), KEY.reducesinkkey2 (type: string),
VALUE._col2 (type: string), VALUE._col3 (type: string), VALUE._col4 (type:
bigint), KEY.reducesinkkey3 (type: string), KEY.reducesinkkey4 (type: string),
KEY.reducesinkkey6 (type: string), KEY.reducesinkkey0 (type: int), VALUE._col6
(type: int), VALUE._col7 (type: double)
outputColumnNames: _col0, _col1, _col3, _col4, _col5, _col6,
_col7, _col9, _col10, _col11, _col12, _col13, _col14
Statistics: Num rows: 579670576 Data size: 591264007848 Basic
stats: COMPLETE Column stats: NONE
PTF Operator
Statistics: Num rows: 579670576 Data size: 591264007848 Basic
stats: COMPLETE Column stats: NONE
Select Operator
expressions: _col3 (type: string), _col4 (type: string),
_col13 (type: int), _col14 (type: double), _col0 (type: int), _wcol1 (type:
int), _col5 (type: string), _col6 (type: string), _col7 (type: bigint), _col9
(type: string), _col10 (type: string), _col11 (type: string)
outputColumnNames: _col0, _col1, _col10, _col11, _col12,
_col13, _col2, _col3, _col4, _col6, _col7, _col8
Statistics: Num rows: 579670576 Data size: 591264007848
Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
key expressions: _col11 (type: double), _col0 (type:
string), _col1 (type: string), _col6 (type: string), _col7 (type: string),
_col11 (type: double), _col0 (type: string), _col1 (type: string), _col6 (type:
string), _col7 (type: string)
sort order: ++++++++++
Map-reduce partition columns: _col11 (type: double),
_col0 (type: string), _col1 (type: string), _col6 (type: string), _col7 (type:
string)
Statistics: Num rows: 579670576 Data size: 591264007848
Basic stats: COMPLETE Column stats: NONE
value expressions: _col2 (type: string), _col3 (type:
string), _col4 (type: bigint), _col8 (type: string), _col10 (type: int), _col12
(type: int), _col13 (type: int)
Reducer 15
Reduce Operator Tree:
Select Operator
expressions: KEY.reducesinkkey1 (type: string),
KEY.reducesinkkey2 (type: string), VALUE._col0 (type: string), VALUE._col1
(type: string), VALUE._col2 (type: bigint), KEY.reducesinkkey3 (type: string),
KEY.reducesinkkey4 (type: string), VALUE._col4 (type: string), VALUE._col6
(type: int), KEY.reducesinkkey0 (type: double), VALUE._col7 (type: int),
VALUE._col8 (type: int)
outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col6,
_col7, _col8, _col10, _col11, _col12, _col13
Statistics: Num rows: 579670576 Data size: 591264007848 Basic
stats: COMPLETE Column stats: NONE
PTF Operator
Statistics: Num rows: 579670576 Data size: 591264007848 Basic
stats: COMPLETE Column stats: NONE
Reduce Output Operator
key expressions: _col11 (type: double), _col0 (type:
string), _col1 (type: string), _col6 (type: string), _col7 (type: string),
_col8 (type: string), _col11 (type: double), _col0 (type: string), _col1 (type:
string), _col6 (type: string), _col7 (type: string), _col8 (type: string)
sort order: ++++++++++++
Map-reduce partition columns: _col11 (type: double), _col0
(type: string), _col1 (type: string), _col6 (type: string), _col7 (type:
string), _col8 (type: string)
Statistics: Num rows: 579670576 Data size: 591264007848
Basic stats: COMPLETE Column stats: NONE
value expressions: _wcol0 (type: int), _col2 (type:
string), _col3 (type: string), _col4 (type: bigint), _col10 (type: int), _col13
(type: int)
Reducer 16
Reduce Operator Tree:
Select Operator
expressions: VALUE._col0 (type: int), KEY.reducesinkkey1 (type:
string), KEY.reducesinkkey2 (type: string), VALUE._col1 (type: string),
VALUE._col2 (type: string), VALUE._col3 (type: bigint), KEY.reducesinkkey3
(type: string), KEY.reducesinkkey4 (type: string), KEY.reducesinkkey5 (type:
string), VALUE._col6 (type: int), KEY.reducesinkkey0 (type: double),
VALUE._col8 (type: int)
outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5,
_col7, _col8, _col9, _col11, _col12, _col14
Statistics: Num rows: 579670576 Data size: 591264007848 Basic
stats: COMPLETE Column stats: NONE
PTF Operator
Statistics: Num rows: 579670576 Data size: 591264007848 Basic
stats: COMPLETE Column stats: NONE
Filter Operator
predicate: (_col0 <= 50) (type: boolean)
Statistics: Num rows: 193223525 Data size: 197088002275
Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: _col1 (type: string), _col2 (type: string),
_col7 (type: string), _col8 (type: string), _wcol1 (type: int), _col9 (type:
string), _col3 (type: string), _col4 (type: string), _col5 (type: bigint)
outputColumnNames: _col0, _col1, _col6, _col7, _col12,
_col8, _col2, _col3, _col4
Statistics: Num rows: 193223525 Data size: 197088002275
Basic stats: COMPLETE Column stats: NONE
Group By Operator
aggregations: count(1), count(DISTINCT _col2),
count(DISTINCT _col2, _col3), max(_col4)
keys: _col0 (type: string), _col1 (type: string), _col6
(type: string), _col7 (type: string), if((_col12 <= 50), _col8, 'Others')
(type: string), _col2 (type: string), _col3 (type: string)
mode: hash
outputColumnNames: _col0, _col1, _col2, _col3, _col4,
_col5, _col6, _col7, _col8, _col9, _col10
Statistics: Num rows: 193223525 Data size: 197088002275
Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
key expressions: _col0 (type: string), _col1 (type:
string), _col2 (type: string), _col3 (type: string), _col4 (type: string),
_col5 (type: string), _col6 (type: string)
sort order: +++++++
Map-reduce partition columns: _col0 (type: string),
_col1 (type: string), _col2 (type: string), _col3 (type: string), _col4 (type:
string)
Statistics: Num rows: 193223525 Data size:
197088002275 Basic stats: COMPLETE Column stats: NONE
value expressions: _col7 (type: bigint), _col10
(type: bigint)
Reducer 17
Reduce Operator Tree:
Group By Operator
aggregations: count(VALUE._col0), count(DISTINCT
KEY._col5:0._col0), count(DISTINCT KEY._col5:1._col0, KEY._col5:1._col1),
max(VALUE._col3)
keys: KEY._col0 (type: string), KEY._col1 (type: string),
KEY._col2 (type: string), KEY._col3 (type: string), KEY._col4 (type: string)
mode: mergepartial
outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5,
_col6, _col7, _col8
Statistics: Num rows: 96611762 Data size: 98544000627 Basic
stats: COMPLETE Column stats: NONE
Select Operator
expressions: _col0 (type: string), '2016-02-03T00:00:00Z'
(type: string), _col1 (type: string), _col2 (type: string), _col3 (type:
string), _col4 (type: string), _col5 (type: bigint), _col6 (type: bigint),
_col7 (type: bigint), _col8 (type: bigint)
outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5,
_col6, _col7, _col8, _col9
Statistics: Num rows: 96611762 Data size: 98544000627 Basic
stats: COMPLETE Column stats: NONE
File Output Operator
compressed: false
Statistics: Num rows: 96611762 Data size: 98544000627 Basic
stats: COMPLETE Column stats: NONE
table:
input format: org.apache.hadoop.mapred.TextInputFormat
output format:
org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
serde:
org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
Reducer 2
Reduce Operator Tree:
Select Operator
expressions: VALUE._col0 (type: string), VALUE._col1 (type:
map<string,string>)
outputColumnNames: _col0, _col1
Statistics: Num rows: 217376466 Data size: 221724002943 Basic
stats: COMPLETE Column stats: NONE
PTF Operator
Statistics: Num rows: 217376466 Data size: 221724002943 Basic
stats: COMPLETE Column stats: NONE
Select Operator
expressions: _col1['meta.appId'] (type: string), _col0
(type: string), _col1['meta.userId'] (type: string), _col1['session_num']
(type: string), _col1 (type: map<string,string>), _wcol0 (type: bigint), _wcol1
(type: int), (_col1['ts'] % 1000) (type: double)
outputColumnNames: _col0, _col1, _col2, _col3, _col4,
_col5, _col6, _col7
Statistics: Num rows: 217376466 Data size: 221724002943
Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
key expressions: _col1 (type: string), _col0 (type:
string), _col6 (type: int), _col1 (type: string), _col0 (type: string), _col6
(type: int)
sort order: ++++++
Map-reduce partition columns: _col1 (type: string), _col0
(type: string), _col6 (type: int)
Statistics: Num rows: 217376466 Data size: 221724002943
Basic stats: COMPLETE Column stats: NONE
value expressions: _col2 (type: string), _col3 (type:
string), _col4 (type: map<string,string>), _col5 (type: bigint), _col7 (type:
double)
Reducer 3
Reduce Operator Tree:
Select Operator
expressions: KEY.reducesinkkey1 (type: string),
KEY.reducesinkkey0 (type: string), VALUE._col0 (type: string), VALUE._col1
(type: string), VALUE._col2 (type: map<string,string>), VALUE._col3 (type:
bigint), KEY.reducesinkkey2 (type: int), VALUE._col4 (type: double)
outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5,
_col6, _col7
Statistics: Num rows: 217376466 Data size: 221724002943 Basic
stats: COMPLETE Column stats: NONE
PTF Operator
Statistics: Num rows: 217376466 Data size: 221724002943 Basic
stats: COMPLETE Column stats: NONE
Select Operator
expressions: _col0 (type: string), _col1 (type: string),
_col2 (type: string), _col3 (type: string), _col4 (type: map<string,string>),
_col6 (type: int), _col7 (type: double), _wcol0 (type: bigint)
outputColumnNames: _col0, _col1, _col2, _col3, _col4,
_col5, _col6, _col7
Statistics: Num rows: 217376466 Data size: 221724002943
Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
key expressions: _col6 (type: double), _col0 (type:
string), _col1 (type: string), _col6 (type: double), _col0 (type: string),
_col1 (type: string)
sort order: ++++++
Map-reduce partition columns: _col6 (type: double), _col0
(type: string), _col1 (type: string)
Statistics: Num rows: 217376466 Data size: 221724002943
Basic stats: COMPLETE Column stats: NONE
value expressions: _col2 (type: string), _col3 (type:
string), _col4 (type: map<string,string>), _col5 (type: int), _col7 (type:
bigint)
Reducer 4
Reduce Operator Tree:
Select Operator
expressions: KEY.reducesinkkey1 (type: string),
KEY.reducesinkkey2 (type: string), VALUE._col0 (type: string), VALUE._col1
(type: string), VALUE._col2 (type: map<string,string>), VALUE._col3 (type:
int), KEY.reducesinkkey0 (type: double), VALUE._col4 (type: bigint)
outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5,
_col6, _col7
Statistics: Num rows: 217376466 Data size: 221724002943 Basic
stats: COMPLETE Column stats: NONE
PTF Operator
Statistics: Num rows: 217376466 Data size: 221724002943 Basic
stats: COMPLETE Column stats: NONE
Select Operator
expressions: _col0 (type: string), _col1 (type: string),
_col2 (type: string), _col3 (type: string), _col4 (type: map<string,string>),
_col6 (type: double), _col5 (type: int), _wcol0 (type: bigint)
outputColumnNames: _col0, _col1, _col2, _col3, _col4,
_col5, _col6, _col7
Statistics: Num rows: 217376466 Data size: 221724002943
Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
key expressions: _col0 (type: string), _col6 (type: int),
_col7 (type: bigint), _col1 (type: string)
sort order: ++-+
Map-reduce partition columns: _col0 (type: string), _col6
(type: int)
Statistics: Num rows: 217376466 Data size: 221724002943
Basic stats: COMPLETE Column stats: NONE
value expressions: _col2 (type: string), _col3 (type:
string), _col4 (type: map<string,string>), _col5 (type: double)
Reducer 5
Reduce Operator Tree:
Select Operator
expressions: KEY.reducesinkkey0 (type: string),
KEY.reducesinkkey3 (type: string), VALUE._col0 (type: string), VALUE._col1
(type: string), VALUE._col2 (type: map<string,string>), VALUE._col3 (type:
double), KEY.reducesinkkey1 (type: int), KEY.reducesinkkey2 (type: bigint)
outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5,
_col6, _col7
Statistics: Num rows: 217376466 Data size: 221724002943 Basic
stats: COMPLETE Column stats: NONE
PTF Operator
Statistics: Num rows: 217376466 Data size: 221724002943 Basic
stats: COMPLETE Column stats: NONE
Select Operator
expressions: _col0 (type: string), _col1 (type: string),
_col2 (type: string), _col3 (type: string), _col4 (type: map<string,string>),
_col5 (type: double), _col7 (type: bigint), _col6 (type: int), _wcol0 (type:
int)
outputColumnNames: _col0, _col1, _col2, _col3, _col4,
_col5, _col6, _col7, _col8
Statistics: Num rows: 217376466 Data size: 221724002943
Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
key expressions: _col5 (type: double), _col0 (type:
string), _col1 (type: string), _col5 (type: double), _col0 (type: string),
_col1 (type: string)
sort order: ++++++
Map-reduce partition columns: _col5 (type: double), _col0
(type: string), _col1 (type: string)
Statistics: Num rows: 217376466 Data size: 221724002943
Basic stats: COMPLETE Column stats: NONE
value expressions: _col2 (type: string), _col3 (type:
string), _col4 (type: map<string,string>), _col6 (type: bigint), _col7 (type:
int), _col8 (type: int)
Reducer 6
Reduce Operator Tree:
Select Operator
expressions: KEY.reducesinkkey1 (type: string),
KEY.reducesinkkey2 (type: string), VALUE._col0 (type: string), VALUE._col1
(type: string), VALUE._col2 (type: map<string,string>), KEY.reducesinkkey0
(type: double), VALUE._col3 (type: bigint), VALUE._col4 (type: int),
VALUE._col5 (type: int)
outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5,
_col6, _col7, _col8
Statistics: Num rows: 217376466 Data size: 221724002943 Basic
stats: COMPLETE Column stats: NONE
PTF Operator
Statistics: Num rows: 217376466 Data size: 221724002943 Basic
stats: COMPLETE Column stats: NONE
Filter Operator
predicate: (_wcol0 <= 150) (type: boolean)
Statistics: Num rows: 72458822 Data size: 73908000981 Basic
stats: COMPLETE Column stats: NONE
Select Operator
expressions: _col0 (type: string), _col1 (type: string),
_col2 (type: string), _col3 (type: string), _col5 (type: double), _col6 (type:
bigint), _col4 (type: map<string,string>)
outputColumnNames: _col0, _col1, _col2, _col3, _col5,
_col6, _col4
Statistics: Num rows: 72458822 Data size: 73908000981
Basic stats: COMPLETE Column stats: NONE
Lateral View Forward
Statistics: Num rows: 72458822 Data size: 73908000981
Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: _col0 (type: string), _col1 (type:
string), _col2 (type: string), _col3 (type: string), _col5 (type: double),
_col6 (type: bigint), _col4 (type: map<string,string>)
outputColumnNames: _col0, _col1, _col2, _col3, _col5,
_col6, _col4
Statistics: Num rows: 72458822 Data size: 73908000981
Basic stats: COMPLETE Column stats: NONE
Lateral View Join Operator
outputColumnNames: _col0, _col1, _col2, _col3,
_col5, _col6, _col4, _col8
Statistics: Num rows: 144917644 Data size:
147816001962 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: _col0 (type: string), _col1 (type:
string), _col2 (type: string), _col3 (type: string), _col5 (type: double),
_col6 (type: bigint), _col8 (type: string), if((_col8 = 'standard'),
to_json(map('Country':_col4['location.country_ip'],'App
Version':_col4['app.version'],'OS Version':_col4['device.os_version'],'Device
Type':_col4['device.type'],'Device
Manufacturer':_col4['device.manufacturer'],'Connection
Type':_col4['device.connection'],'Device
Carrier':_col4['device.carrier'],'Language':_col4['locale'],'Device
OS':_col4['device.os'])), _col4[_col8]) (type: string)
outputColumnNames: _col0, _col1, _col2, _col3,
_col4, _col5, _col7, _col8
Statistics: Num rows: 144917644 Data size:
147816001962 Basic stats: COMPLETE Column stats: NONE
Lateral View Forward
Statistics: Num rows: 144917644 Data size:
147816001962 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: _col0 (type: string), _col1
(type: string), _col2 (type: string), _col3 (type: string), _col4 (type:
double), _col5 (type: bigint), _col7 (type: string)
outputColumnNames: _col0, _col1, _col2,
_col3, _col4, _col5, _col7
Statistics: Num rows: 144917644 Data size:
147816001962 Basic stats: COMPLETE Column stats: NONE
Lateral View Join Operator
outputColumnNames: _col0, _col1, _col2,
_col3, _col4, _col5, _col7, _col9, _col10, _col11, _col12, _col13, _col14,
_col15
Statistics: Num rows: 289835288 Data size:
295632003924 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: _col0 (type: string), _col1
(type: string), _col2 (type: string), _col3 (type: string), _col4 (type:
double), _col5 (type: bigint), _col7 (type: string), _col9 (type:
map<string,string>)
outputColumnNames: _col0, _col1, _col2,
_col3, _col4, _col5, _col7, _col8
Statistics: Num rows: 289835288 Data
size: 295632003924 Basic stats: COMPLETE Column stats: NONE
Lateral View Forward
Statistics: Num rows: 289835288 Data
size: 295632003924 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: _col0 (type: string),
_col1 (type: string), _col2 (type: string), _col3 (type: string), _col4 (type:
double), _col5 (type: bigint), _col7 (type: string)
outputColumnNames: _col0, _col1,
_col2, _col3, _col4, _col5, _col7
Statistics: Num rows: 289835288 Data
size: 295632003924 Basic stats: COMPLETE Column stats: NONE
Lateral View Join Operator
outputColumnNames: _col0, _col1,
_col2, _col3, _col4, _col5, _col7, _col9, _col10
Statistics: Num rows: 579670576
Data size: 591264007848 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: _col0 (type:
string), _col1 (type: string), _col2 (type: string), _col3 (type: string),
_col4 (type: double), _col5 (type: bigint), _col7 (type: string), _col9 (type:
string), _col10 (type: string)
outputColumnNames: _col0, _col1,
_col2, _col3, _col4, _col5, _col7, _col8, _col9
Statistics: Num rows: 579670576
Data size: 591264007848 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
key expressions: _col0 (type:
string), _col1 (type: string), _col7 (type: string), _col8 (type: string),
_col9 (type: string), _col4 (type: double), _col0 (type: string), _col1 (type:
string), _col7 (type: string), _col8 (type: string), _col9 (type: string),
_col4 (type: double)
sort order: ++++++++++++
Map-reduce partition columns:
_col0 (type: string), _col1 (type: string), _col7 (type: string), _col8 (type:
string), _col9 (type: string), _col4 (type: double)
Statistics: Num rows: 579670576
Data size: 591264007848 Basic stats: COMPLETE Column stats: NONE
value expressions: _col2 (type:
string), _col3 (type: string), _col5 (type: bigint)
Select Operator
expressions: _col8 (type:
map<string,string>)
outputColumnNames: _col0
Statistics: Num rows: 289835288 Data
size: 295632003924 Basic stats: COMPLETE Column stats: NONE
UDTF Operator
Statistics: Num rows: 289835288
Data size: 295632003924 Basic stats: COMPLETE Column stats: NONE
function name: explode
Lateral View Join Operator
outputColumnNames: _col0, _col1,
_col2, _col3, _col4, _col5, _col7, _col9, _col10
Statistics: Num rows: 579670576
Data size: 591264007848 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: _col0 (type:
string), _col1 (type: string), _col2 (type: string), _col3 (type: string),
_col4 (type: double), _col5 (type: bigint), _col7 (type: string), _col9 (type:
string), _col10 (type: string)
outputColumnNames: _col0,
_col1, _col2, _col3, _col4, _col5, _col7, _col8, _col9
Statistics: Num rows: 579670576
Data size: 591264007848 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
key expressions: _col0 (type:
string), _col1 (type: string), _col7 (type: string), _col8 (type: string),
_col9 (type: string), _col4 (type: double), _col0 (type: string), _col1 (type:
string), _col7 (type: string), _col8 (type: string), _col9 (type: string),
_col4 (type: double)
sort order: ++++++++++++
Map-reduce partition columns:
_col0 (type: string), _col1 (type: string), _col7 (type: string), _col8 (type:
string), _col9 (type: string), _col4 (type: double)
Statistics: Num rows:
579670576 Data size: 591264007848 Basic stats: COMPLETE Column stats: NONE
value expressions: _col2
(type: string), _col3 (type: string), _col5 (type: bigint)
Select Operator
expressions: 'map<string,string>' (type:
string), 'pub_data' (type: string), map('pub_data':_col8) (type:
map<string,string>), '0000000000' (type: string)
outputColumnNames: _col0, _col1, _col2, _col3
Statistics: Num rows: 144917644 Data size:
147816001962 Basic stats: COMPLETE Column stats: NONE
UDTF Operator
Statistics: Num rows: 144917644 Data size:
147816001962 Basic stats: COMPLETE Column stats: NONE
function name: up_explode_map
Lateral View Join Operator
outputColumnNames: _col0, _col1, _col2,
_col3, _col4, _col5, _col7, _col9, _col10, _col11, _col12, _col13, _col14,
_col15
Statistics: Num rows: 289835288 Data
size: 295632003924 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: _col0 (type: string),
_col1 (type: string), _col2 (type: string), _col3 (type: string), _col4 (type:
double), _col5 (type: bigint), _col7 (type: string), _col9 (type:
map<string,string>)
outputColumnNames: _col0, _col1, _col2,
_col3, _col4, _col5, _col7, _col8
Statistics: Num rows: 289835288 Data
size: 295632003924 Basic stats: COMPLETE Column stats: NONE
Lateral View Forward
Statistics: Num rows: 289835288 Data
size: 295632003924 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: _col0 (type: string),
_col1 (type: string), _col2 (type: string), _col3 (type: string), _col4 (type:
double), _col5 (type: bigint), _col7 (type: string)
outputColumnNames: _col0, _col1,
_col2, _col3, _col4, _col5, _col7
Statistics: Num rows: 289835288
Data size: 295632003924 Basic stats: COMPLETE Column stats: NONE
Lateral View Join Operator
outputColumnNames: _col0, _col1,
_col2, _col3, _col4, _col5, _col7, _col9, _col10
Statistics: Num rows: 579670576
Data size: 591264007848 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: _col0 (type:
string), _col1 (type: string), _col2 (type: string), _col3 (type: string),
_col4 (type: double), _col5 (type: bigint), _col7 (type: string), _col9 (type:
string), _col10 (type: string)
outputColumnNames: _col0,
_col1, _col2, _col3, _col4, _col5, _col7, _col8, _col9
Statistics: Num rows: 579670576
Data size: 591264007848 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
key expressions: _col0 (type:
string), _col1 (type: string), _col7 (type: string), _col8 (type: string),
_col9 (type: string), _col4 (type: double), _col0 (type: string), _col1 (type:
string), _col7 (type: string), _col8 (type: string), _col9 (type: string),
_col4 (type: double)
sort order: ++++++++++++
Map-reduce partition columns:
_col0 (type: string), _col1 (type: string), _col7 (type: string), _col8 (type:
string), _col9 (type: string), _col4 (type: double)
Statistics: Num rows:
579670576 Data size: 591264007848 Basic stats: COMPLETE Column stats: NONE
value expressions: _col2
(type: string), _col3 (type: string), _col5 (type: bigint)
Select Operator
expressions: _col8 (type:
map<string,string>)
outputColumnNames: _col0
Statistics: Num rows: 289835288
Data size: 295632003924 Basic stats: COMPLETE Column stats: NONE
UDTF Operator
Statistics: Num rows: 289835288
Data size: 295632003924 Basic stats: COMPLETE Column stats: NONE
function name: explode
Lateral View Join Operator
outputColumnNames: _col0,
_col1, _col2, _col3, _col4, _col5, _col7, _col9, _col10
Statistics: Num rows: 579670576
Data size: 591264007848 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: _col0 (type:
string), _col1 (type: string), _col2 (type: string), _col3 (type: string),
_col4 (type: double), _col5 (type: bigint), _col7 (type: string), _col9 (type:
string), _col10 (type: string)
outputColumnNames: _col0,
_col1, _col2, _col3, _col4, _col5, _col7, _col8, _col9
Statistics: Num rows:
579670576 Data size: 591264007848 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
key expressions: _col0
(type: string), _col1 (type: string), _col7 (type: string), _col8 (type:
string), _col9 (type: string), _col4 (type: double), _col0 (type: string),
_col1 (type: string), _col7 (type: string), _col8 (type: string), _col9 (type:
string), _col4 (type: double)
sort order: ++++++++++++
Map-reduce partition
columns: _col0 (type: string), _col1 (type: string), _col7 (type: string),
_col8 (type: string), _col9 (type: string), _col4 (type: double)
Statistics: Num rows:
579670576 Data size: 591264007848 Basic stats: COMPLETE Column stats: NONE
value expressions: _col2
(type: string), _col3 (type: string), _col5 (type: bigint)
Select Operator
expressions:
array('pub_data','user_attributes','standard') (type: array<string>)
outputColumnNames: _col0
Statistics: Num rows: 72458822 Data size: 73908000981
Basic stats: COMPLETE Column stats: NONE
UDTF Operator
Statistics: Num rows: 72458822 Data size:
73908000981 Basic stats: COMPLETE Column stats: NONE
function name: explode
Lateral View Join Operator
outputColumnNames: _col0, _col1, _col2, _col3,
_col5, _col6, _col4, _col8
Statistics: Num rows: 144917644 Data size:
147816001962 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: _col0 (type: string), _col1 (type:
string), _col2 (type: string), _col3 (type: string), _col5 (type: double),
_col6 (type: bigint), _col8 (type: string), if((_col8 = 'standard'),
to_json(map('Country':_col4['location.country_ip'],'App
Version':_col4['app.version'],'OS Version':_col4['device.os_version'],'Device
Type':_col4['device.type'],'Device
Manufacturer':_col4['device.manufacturer'],'Connection
Type':_col4['device.connection'],'Device
Carrier':_col4['device.carrier'],'Language':_col4['locale'],'Device
OS':_col4['device.os'])), _col4[_col8]) (type: string)
outputColumnNames: _col0, _col1, _col2, _col3,
_col4, _col5, _col7, _col8
Statistics: Num rows: 144917644 Data size:
147816001962 Basic stats: COMPLETE Column stats: NONE
Lateral View Forward
Statistics: Num rows: 144917644 Data size:
147816001962 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: _col0 (type: string), _col1
(type: string), _col2 (type: string), _col3 (type: string), _col4 (type:
double), _col5 (type: bigint), _col7 (type: string)
outputColumnNames: _col0, _col1, _col2,
_col3, _col4, _col5, _col7
Statistics: Num rows: 144917644 Data size:
147816001962 Basic stats: COMPLETE Column stats: NONE
Lateral View Join Operator
outputColumnNames: _col0, _col1, _col2,
_col3, _col4, _col5, _col7, _col9, _col10, _col11, _col12, _col13, _col14,
_col15
Statistics: Num rows: 289835288 Data
size: 295632003924 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: _col0 (type: string),
_col1 (type: string), _col2 (type: string), _col3 (type: string), _col4 (type:
double), _col5 (type: bigint), _col7 (type: string), _col9 (type:
map<string,string>)
outputColumnNames: _col0, _col1, _col2,
_col3, _col4, _col5, _col7, _col8
Statistics: Num rows: 289835288 Data
size: 295632003924 Basic stats: COMPLETE Column stats: NONE
Lateral View Forward
Statistics: Num rows: 289835288 Data
size: 295632003924 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: _col0 (type: string),
_col1 (type: string), _col2 (type: string), _col3 (type: string), _col4 (type:
double), _col5 (type: bigint), _col7 (type: string)
outputColumnNames: _col0, _col1,
_col2, _col3, _col4, _col5, _col7
Statistics: Num rows: 289835288
Data size: 295632003924 Basic stats: COMPLETE Column stats: NONE
Lateral View Join Operator
outputColumnNames: _col0, _col1,
_col2, _col3, _col4, _col5, _col7, _col9, _col10
Statistics: Num rows: 579670576
Data size: 591264007848 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: _col0 (type:
string), _col1 (type: string), _col2 (type: string), _col3 (type: string),
_col4 (type: double), _col5 (type: bigint), _col7 (type: string), _col9 (type:
string), _col10 (type: string)
outputColumnNames: _col0,
_col1, _col2, _col3, _col4, _col5, _col7, _col8, _col9
Statistics: Num rows: 579670576
Data size: 591264007848 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
key expressions: _col0 (type:
string), _col1 (type: string), _col7 (type: string), _col8 (type: string),
_col9 (type: string), _col4 (type: double), _col0 (type: string), _col1 (type:
string), _col7 (type: string), _col8 (type: string), _col9 (type: string),
_col4 (type: double)
sort order: ++++++++++++
Map-reduce partition columns:
_col0 (type: string), _col1 (type: string), _col7 (type: string), _col8 (type:
string), _col9 (type: string), _col4 (type: double)
Statistics: Num rows:
579670576 Data size: 591264007848 Basic stats: COMPLETE Column stats: NONE
value expressions: _col2
(type: string), _col3 (type: string), _col5 (type: bigint)
Select Operator
expressions: _col8 (type:
map<string,string>)
outputColumnNames: _col0
Statistics: Num rows: 289835288
Data size: 295632003924 Basic stats: COMPLETE Column stats: NONE
UDTF Operator
Statistics: Num rows: 289835288
Data size: 295632003924 Basic stats: COMPLETE Column stats: NONE
function name: explode
Lateral View Join Operator
outputColumnNames: _col0,
_col1, _col2, _col3, _col4, _col5, _col7, _col9, _col10
Statistics: Num rows: 579670576
Data size: 591264007848 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: _col0 (type:
string), _col1 (type: string), _col2 (type: string), _col3 (type: string),
_col4 (type: double), _col5 (type: bigint), _col7 (type: string), _col9 (type:
string), _col10 (type: string)
outputColumnNames: _col0,
_col1, _col2, _col3, _col4, _col5, _col7, _col8, _col9
Statistics: Num rows:
579670576 Data size: 591264007848 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
key expressions: _col0
(type: string), _col1 (type: string), _col7 (type: string), _col8 (type:
string), _col9 (type: string), _col4 (type: double), _col0 (type: string),
_col1 (type: string), _col7 (type: string), _col8 (type: string), _col9 (type:
string), _col4 (type: double)
sort order: ++++++++++++
Map-reduce partition
columns: _col0 (type: string), _col1 (type: string), _col7 (type: string),
_col8 (type: string), _col9 (type: string), _col4 (type: double)
Statistics: Num rows:
579670576 Data size: 591264007848 Basic stats: COMPLETE Column stats: NONE
value expressions: _col2
(type: string), _col3 (type: string), _col5 (type: bigint)
Select Operator
expressions: 'map<string,string>' (type:
string), 'pub_data' (type: string), map('pub_data':_col8) (type:
map<string,string>), '0000000000' (type: string)
outputColumnNames: _col0, _col1, _col2,
_col3
Statistics: Num rows: 144917644 Data size:
147816001962 Basic stats: COMPLETE Column stats: NONE
UDTF Operator
Statistics: Num rows: 144917644 Data
size: 147816001962 Basic stats: COMPLETE Column stats: NONE
function name: up_explode_map
Lateral View Join Operator
outputColumnNames: _col0, _col1, _col2,
_col3, _col4, _col5, _col7, _col9, _col10, _col11, _col12, _col13, _col14,
_col15
Statistics: Num rows: 289835288 Data
size: 295632003924 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: _col0 (type: string),
_col1 (type: string), _col2 (type: string), _col3 (type: string), _col4 (type:
double), _col5 (type: bigint), _col7 (type: string), _col9 (type:
map<string,string>)
outputColumnNames: _col0, _col1,
_col2, _col3, _col4, _col5, _col7, _col8
Statistics: Num rows: 289835288 Data
size: 295632003924 Basic stats: COMPLETE Column stats: NONE
Lateral View Forward
Statistics: Num rows: 289835288
Data size: 295632003924 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: _col0 (type:
string), _col1 (type: string), _col2 (type: string), _col3 (type: string),
_col4 (type: double), _col5 (type: bigint), _col7 (type: string)
outputColumnNames: _col0, _col1,
_col2, _col3, _col4, _col5, _col7
Statistics: Num rows: 289835288
Data size: 295632003924 Basic stats: COMPLETE Column stats: NONE
Lateral View Join Operator
outputColumnNames: _col0,
_col1, _col2, _col3, _col4, _col5, _col7, _col9, _col10
Statistics: Num rows: 579670576
Data size: 591264007848 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: _col0 (type:
string), _col1 (type: string), _col2 (type: string), _col3 (type: string),
_col4 (type: double), _col5 (type: bigint), _col7 (type: string), _col9 (type:
string), _col10 (type: string)
outputColumnNames: _col0,
_col1, _col2, _col3, _col4, _col5, _col7, _col8, _col9
Statistics: Num rows:
579670576 Data size: 591264007848 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
key expressions: _col0
(type: string), _col1 (type: string), _col7 (type: string), _col8 (type:
string), _col9 (type: string), _col4 (type: double), _col0 (type: string),
_col1 (type: string), _col7 (type: string), _col8 (type: string), _col9 (type:
string), _col4 (type: double)
sort order: ++++++++++++
Map-reduce partition
columns: _col0 (type: string), _col1 (type: string), _col7 (type: string),
_col8 (type: string), _col9 (type: string), _col4 (type: double)
Statistics: Num rows:
579670576 Data size: 591264007848 Basic stats: COMPLETE Column stats: NONE
value expressions: _col2
(type: string), _col3 (type: string), _col5 (type: bigint)
Select Operator
expressions: _col8 (type:
map<string,string>)
outputColumnNames: _col0
Statistics: Num rows: 289835288
Data size: 295632003924 Basic stats: COMPLETE Column stats: NONE
UDTF Operator
Statistics: Num rows: 289835288
Data size: 295632003924 Basic stats: COMPLETE Column stats: NONE
function name: explode
Lateral View Join Operator
outputColumnNames: _col0,
_col1, _col2, _col3, _col4, _col5, _col7, _col9, _col10
Statistics: Num rows:
579670576 Data size: 591264007848 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: _col0 (type:
string), _col1 (type: string), _col2 (type: string), _col3 (type: string),
_col4 (type: double), _col5 (type: bigint), _col7 (type: string), _col9 (type:
string), _col10 (type: string)
outputColumnNames: _col0,
_col1, _col2, _col3, _col4, _col5, _col7, _col8, _col9
Statistics: Num rows:
579670576 Data size: 591264007848 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
key expressions: _col0
(type: string), _col1 (type: string), _col7 (type: string), _col8 (type:
string), _col9 (type: string), _col4 (type: double), _col0 (type: string),
_col1 (type: string), _col7 (type: string), _col8 (type: string), _col9 (type:
string), _col4 (type: double)
sort order: ++++++++++++
Map-reduce partition
columns: _col0 (type: string), _col1 (type: string), _col7 (type: string),
_col8 (type: string), _col9 (type: string), _col4 (type: double)
Statistics: Num rows:
579670576 Data size: 591264007848 Basic stats: COMPLETE Column stats: NONE
value expressions: _col2
(type: string), _col3 (type: string), _col5 (type: bigint)
Reducer 7
Reduce Operator Tree:
Select Operator
expressions: KEY.reducesinkkey0 (type: string),
KEY.reducesinkkey1 (type: string), VALUE._col0 (type: string), VALUE._col1
(type: string), KEY.reducesinkkey5 (type: double), VALUE._col2 (type: bigint),
KEY.reducesinkkey2 (type: string), KEY.reducesinkkey3 (type: string),
KEY.reducesinkkey4 (type: string)
outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5,
_col7, _col8, _col9
Statistics: Num rows: 579670576 Data size: 591264007848 Basic
stats: COMPLETE Column stats: NONE
PTF Operator
Statistics: Num rows: 579670576 Data size: 591264007848 Basic
stats: COMPLETE Column stats: NONE
Reduce Output Operator
key expressions: _col0 (type: string), _col1 (type:
string), _col7 (type: string), _col8 (type: string), _col4 (type: double),
_col0 (type: string), _col1 (type: string), _col7 (type: string), _col8 (type:
string), _col4 (type: double)
sort order: ++++++++++
Map-reduce partition columns: _col0 (type: string), _col1
(type: string), _col7 (type: string), _col8 (type: string), _col4 (type: double)
Statistics: Num rows: 579670576 Data size: 591264007848
Basic stats: COMPLETE Column stats: NONE
value expressions: _wcol0 (type: bigint), _wcol1 (type:
int), _col2 (type: string), _col3 (type: string), _col5 (type: bigint), _col9
(type: string)
Reducer 8
Reduce Operator Tree:
Select Operator
expressions: VALUE._col0 (type: bigint), VALUE._col1 (type:
int), KEY.reducesinkkey0 (type: string), KEY.reducesinkkey1 (type: string),
VALUE._col2 (type: string), VALUE._col3 (type: string), KEY.reducesinkkey4
(type: double), VALUE._col4 (type: bigint), KEY.reducesinkkey2 (type: string),
KEY.reducesinkkey3 (type: string), VALUE._col6 (type: string)
outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5,
_col6, _col7, _col9, _col10, _col11
Statistics: Num rows: 579670576 Data size: 591264007848 Basic
stats: COMPLETE Column stats: NONE
PTF Operator
Statistics: Num rows: 579670576 Data size: 591264007848 Basic
stats: COMPLETE Column stats: NONE
Select Operator
expressions: _col0 (type: bigint), _col1 (type: int), _col9
(type: string), _col10 (type: string), _col11 (type: string), _col6 (type:
double), _wcol2 (type: bigint), _wcol3 (type: int), _col2 (type: string), _col3
(type: string), _col4 (type: string), _col5 (type: string), _col7 (type: bigint)
outputColumnNames: _col0, _col1, _col10, _col11, _col12,
_col13, _col2, _col3, _col4, _col5, _col6, _col7, _col8
Statistics: Num rows: 579670576 Data size: 591264007848
Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
key expressions: _col4 (type: string), _col5 (type:
string), _col10 (type: string), _col11 (type: string), _col12 (type: string),
_col1 (type: int), _col4 (type: string), _col5 (type: string), _col10 (type:
string), _col11 (type: string), _col12 (type: string), _col1 (type: int)
sort order: ++++++++++++
Map-reduce partition columns: _col4 (type: string), _col5
(type: string), _col10 (type: string), _col11 (type: string), _col12 (type:
string), _col1 (type: int)
Statistics: Num rows: 579670576 Data size: 591264007848
Basic stats: COMPLETE Column stats: NONE
value expressions: _col0 (type: bigint), _col2 (type:
bigint), _col3 (type: int), _col6 (type: string), _col7 (type: string), _col8
(type: bigint), _col13 (type: double)
Reducer 9
Reduce Operator Tree:
Select Operator
expressions: VALUE._col0 (type: bigint), KEY.reducesinkkey5
(type: int), VALUE._col1 (type: bigint), VALUE._col2 (type: int),
KEY.reducesinkkey0 (type: string), KEY.reducesinkkey1 (type: string),
VALUE._col3 (type: string), VALUE._col4 (type: string), VALUE._col5 (type:
bigint), KEY.reducesinkkey2 (type: string), KEY.reducesinkkey3 (type: string),
KEY.reducesinkkey4 (type: string), VALUE._col7 (type: double)
outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5,
_col6, _col7, _col8, _col10, _col11, _col12, _col13
Statistics: Num rows: 579670576 Data size: 591264007848 Basic
stats: COMPLETE Column stats: NONE
PTF Operator
Statistics: Num rows: 579670576 Data size: 591264007848 Basic
stats: COMPLETE Column stats: NONE
Reduce Output Operator
key expressions: _col4 (type: string), _col5 (type:
string), _col10 (type: string), _col11 (type: string), _col3 (type: int), _col4
(type: string), _col5 (type: string), _col10 (type: string), _col11 (type:
string), _col3 (type: int)
sort order: ++++++++++
Map-reduce partition columns: _col4 (type: string), _col5
(type: string), _col10 (type: string), _col11 (type: string), _col3 (type: int)
Statistics: Num rows: 579670576 Data size: 591264007848
Basic stats: COMPLETE Column stats: NONE
value expressions: _wcol0 (type: bigint), _col1 (type:
int), _col2 (type: bigint), _col6 (type: string), _col7 (type: string), _col8
(type: bigint), _col12 (type: string), _col13 (type: double)
Stage: Stage-0
Fetch Operator
limit: -1
Processor Tree:
ListSink
Time taken: 1.351 seconds, Fetched: 559 row(s)
