Re: Tez reducer parallelism ..

2016-03-19 Thread Gopal Vijayaraghavan
> So you'r saying, since these windows are part of a single SELECT >projection they need to be serial? Yes, with a full shuffle of the result so far for each new OVER(). > row_number() OVER( PARTITION BY app, user, type ORDER BY ts >) as a_number, > row_number() OVER( P

Re: Tez reducer parallelism ..

2016-03-15 Thread Gautam
> The windowing is not simultaneous unless they are all over the same window > - the following query has 3 different windows applied over the same rows > sequentially. Ok. Just wanted to confirm. Maybe I could restructure my query to get more parallelism .. > They are all over the same rows so th

Re: Tez reducer parallelism ..

2016-03-15 Thread Gopal Vijayaraghavan
> A lot of our queries do the following style of simultaneous windowing .. The windowing is not simultaneous unless they are all over the same window - the following query has 3 different windows applied over the same rows sequentially. > SELECT >row_number() OVER( PARTITION BY app, user, > t

Tez reducer parallelism ..

2016-03-15 Thread Gautam
Hello, I'm trying to optimize some queries in Hive that were recently switched to Tez.. had a general question regarding reducer parallelism .. A lot of our queries do the following style of simultaneous windowing .. SELECT row_number() OVER( PARTITION BY app, user, type ORDER BY ts ) as