Hi Mark, Thanks for your response. I tried skew optimization and I also saw the video by Lin and Namit. From what I understand about skew join, instead of a single go , they divide it into 2 stages.
Stage1 Join non-skew pairs. and write the skew pairs into temporary files on HDFS. Stage 2 Do a Map-Join of the files by copying smaller file into mappers of larger file. I have a doubt here. How can they be so sure that MapJoin works in stage 2? The files can be so large that they donot fit into the memory and join is impossible. Am I wrong? I also ran the query with skew optimized and as expected, none of the the pairs got joined in the stage 1 and all of them got written into the HDFS. (They are huge) Now in the stage2 , Hive is trying to perform a map-join on these large tables and my Map phase in stage 2 is stuck at 0.13% after 6 hours and 2 of my machines went down. I had to kill the job finally. The size of each table is just 2GB which is way smaller than what Hadoop eco system can handle. So is there anyway I can join these tables in Hive? Any thoughts ? Thanks, jS On Tue, Dec 6, 2011 at 3:39 AM, Mark Grover <mgro...@oanda.com> wrote: > jS, > Check out if this helps: > > http://search-hadoop.com/m/l1usr1MAHX32&subj=Re+Severely+hit+by+curse+of+last+reducer+ > > > > Mark Grover, Business Intelligence Analyst > OANDA Corporation > > www: oanda.com www: fxtrade.com > e: mgro...@oanda.com > > "Best Trading Platform" - World Finance's Forex Awards 2009. > "The One to Watch" - Treasury Today's Adam Smith Awards 2009. > > > ----- Original Message ----- > From: "john smith" <js1987.sm...@gmail.com> > To: user@hive.apache.org > Sent: Monday, December 5, 2011 4:38:14 PM > Subject: Hive Reducers hanging - interesting problem - skew ? > > Hi list, > > I am trying to run a Join query on my 10 node cluster. My query looks as > follows > > select * from A JOIN B on (A.a = B.b) > > size of A = 15 million rows > size of B = 1 million rows > > The problem is A.a and B.b has around 25-30 distinct values per column > which implies that they have high selectivities and the reducers are bulky. > > However the performance hit is so horrible that , ALL my reducers hang @ > 75% for 6 hours and doesn't move further. > > The only thing that log shows up is "Join operator - forwarding rows > ---------------<Huge number>" kinds of logs for all this long. What does > this mean ? > There is no swapping happening and the CPU % is constantly around 40% for > all this time (observed through Ganglia) . > > Any way I can solve this problem? Can anyone help me with this? > > Thanks, > jS > > >
