Re: Setting number of parallel Reducers and Mappers for optimal performance

2012-08-11 Thread Arun C Murthy
Pavan, On Aug 10, 2012, at 9:17 PM, Pavan Kulkarni wrote: > Arun, > > Thanks a lot for your response. > > I am running on a 16 core Xeon processor and 12 spindles.So running 12 > Mappers with 2G and 6 Reducers with 3G might give me the best > performance. Hmm... ok. You actually _may_ have e

Re: Setting number of parallel Reducers and Mappers for optimal performance

2012-08-10 Thread Pavan Kulkarni
Arun, Thanks a lot for your response. I am running on a 16 core Xeon processor and 12 spindles.So running 12 Mappers with 2G and 6 Reducers with 3G might give me the best performance.Also is there a general formula to arrive at those numbers? On Fri, Aug 10, 2012 at 7:34 PM, Arun C Murthy wro

Re: Setting number of parallel Reducers and Mappers for optimal performance

2012-08-10 Thread Arun C Murthy
Pavan, A very important factor is how much CPU and how many spindles you have... Your proposal for memory (44G in all) seems reasonable. However, if you have 12 spindles and sufficient CPU I'd do something like 10 or 12 maps of 2G each and 6 reduces with 3G/4G each depending on how you want to