Does the execution plan for the query give any hints? In my case I noticed data was picked from the buckets using a hashkey. Subsequently the workload was split between reducers using the same hashkey. So, only one reducer ended up doing all the work. I don't know how to fix this problem yet ... but it sounds like that is not the problem you have.
Cheers, -Ajo. On Thu, Feb 10, 2011 at 4:55 PM, Viral Bajaria <viral.baja...@gmail.com>wrote: > there were 3 different queries which exhibited this behavior ... one was > over 30-days worth of data and 2 were over 7-days worth of data. > > > On Thu, Feb 10, 2011 at 3:49 PM, Jonathan Coveney <jcove...@gmail.com>wrote: > >> How many days of data are you working on? >> >> >> Sent via BlackBerry >> ------------------------------ >> *From: * Viral Bajaria <viral.baja...@gmail.com> >> *Date: *Thu, 10 Feb 2011 15:21:32 -0800 >> *To: *<user@hive.apache.org> >> *ReplyTo: * user@hive.apache.org >> *Subject: *Re: hive : question about reducers >> >> I don't have any explicit bucketing in my data. The data is partitioned by >> current_date (it has no hour information, so basically 24 hours of data). >> >> It's not a problem because eventually the job would complete (super-slow) >> but it would be nice to know the reason behind this behavior and how I could >> optimize it so that I can take full advantage of having multiple reducers >> running. >> >> -Viral >> >> On Thu, Feb 10, 2011 at 3:02 PM, Ajo Fod <ajo....@gmail.com> wrote: >> >>> I've had similar experiences ... usually with bucketing. >>> >>> Is this your experience too? >>> >>> -Ajo >>> >>> >>> On Thu, Feb 10, 2011 at 1:57 PM, Viral Bajaria >>> <viral.baja...@gmail.com>wrote: >>> >>>> Hello, >>>> >>>> In my Hive cluster, I have setup the mapred.reduce.tasks to be -1 i.e. I >>>> am allowing HIVE to figure out the # of reducers that it would need from >>>> the >>>> data. >>>> >>>> When I run a query, it determines that it will need 4 reducers but when >>>> I look at the MAPRED logs, I see that all the work is done by a single >>>> reducer while the other 3 reducers forward 0 rows. Is this just bad >>>> planning >>>> on HIVE side or am I missing something. >>>> >>>> Thanks, >>>> Viral >>>> >>> >>> >> >
