Hi Abshiek Based on my experience you can always provide the number of reduce tasks (mapred.reduce.tasks) based on the data volume your query handles. It can yield you better performance numbers. Regards, Bejoy KS
________________________________ From: Abhishek <abhishek.dod...@gmail.com> To: "user@hive.apache.org" <user@hive.apache.org> Cc: "user@hive.apache.org" <user@hive.apache.org> Sent: Wednesday, September 26, 2012 7:04 PM Subject: Re: Hive configuration property Thanks Bharath, Your points make sense.I'll try this "hive.exec.reducers.max" property. Regards Abhi Sent from my iPhone On Sep 26, 2012, at 9:23 AM, bharath vissapragada <bharathvissapragada1...@gmail.com> wrote: > >I'm no expert in hive, but here are my 2 cents. > > >By default hive schedules a reducer per every 1 GB of data ( change that value >by modifying hive.exec.reducers.bytes.per.reducer ) . If your input data is >huge, there will be large number of reducers, which might be unnecessary.( >Sometimes large number of reducers slows down the job because their number >exceeds total task slots and they keep waiting for their turn. Not to forget, >the initialization overheads for each task..jvm etc.). > > >Overall, I think there cannot be any optimum values for a cluster. It depends >on the type of queries, size of your inputs, size of map outputs in the jobs >(intermediate outputs ). So you can can check various values and see which one >is the best. From my experience setting "hive.exec.reducers.max" to total >number of reduce slots in your cluster gives you a decent performance since >all the reducers are completed in a single wave. (This may or maynot work for >you, worth giving a try). > > > > >On Wed, Sep 26, 2012 at 5:58 PM, Abhishek <abhishek.dod...@gmail.com> wrote: > > >> >>Hi all, >> >> >>I have doubt regarding below properties, is it a good practice to override >>below properties in hive. >> >> >>If yes, what is the optimal values for the following properties? >> >> set hive.exec.reducers.bytes.per.reducer=<number> >>In order to limit the maximum number of reducers: >> set hive.exec.reducers.max=<number> >>In order to set a constant number of reducers: >> set mapred.reduce.tasks=<number> >> >> >>Regards >>Abhi >> >>Sent from my iPhone > > > >-- >Regards, >Bharath .V >w:http://researchweb.iiit.ac.in/~bharath.v >
