That is what I was wondering. I would expect the join and the aggregate to be combined in 1 job much faster as it would save storing results and reading them back in ... Is there a way (hint) to tell hive to do so ?
From: Sudhish Iyer <sudh...@yahoo-inc.com> To: "user@hive.apache.org" <user@hive.apache.org> Date: 03/24/2011 05:02 PM Subject: RE: Number of map reduce jobs generated In this case would 2 M/R job run faster than one? From: Tim Kaldewey [mailto:tkal...@us.ibm.com] Sent: Thursday, March 24, 2011 2:30 PM To: user@hive.apache.org Subject: Number of map reduce jobs generated Hello, I noticed that for join queries that comprise an aggregate hive generates a query plan with two MR jobs, one does the join and the second the aggregate. I was wondering if there is a way to hint hive to combine these two operations in 1 MR job. I have attached an example of the set of queries I am looking at. Thanks Tim select /*+ MAPJOIN(Table2) */ sum(t1_10 * t1_12) from Table1 join Table2 on (Table1.t1_6 = Table2.t2_1) where Table2.t2_5 = 1234 and 8 <= Table1.t1_12 <= 10 and Table1.t1_9 < 42; to explain: - table 2 is small, thus I choose a map-side (broadcast) join. - when I remove the aggregate hive only generates 1MR job
<<inline: graycol.gif>>
