We just update the slides of this improvements:
http://www.slideshare.net/hanborq/hanborq-optimizations-on-hadoop-mapreduce-20120216a
Updates:
(1) modified some describes to make things more clear and accuracy.
(2) add some benchmarks to make sense.
On Sat, Feb 18, 2012 at 11:12 PM, Anty wrote:
On Fri, Feb 17, 2012 at 3:27 AM, Todd Lipcon wrote:
> Hey Schubert,
>
> Looking at the code on github, it looks like your rewritten shuffle is
> in fact just a backport of the shuffle from MR2. I didn't look closely
>
additionally, the rewritten shuffle in MR2 has some bugs, which harm the
overa
On Thu, Feb 16, 2012 at 8:25 PM, Schubert Zhang wrote:
> 1) it should be sort-avoidance.
right - that's a nice improvement, looking forward to getting that in
trunk at some point.
> 2) work pool (like Tenzing)
>
Looking at the code, it seems you only support the default task
executor. Do you ha
1) it should be sort-avoidance.
2) work pool (like Tenzing)
Sorry ,the adaptive heartbeat code is not in this github code, we are
discussing it.
On Fri, Feb 17, 2012 at 11:00 AM, Anty wrote:
> Hi: Todd
>
> yes, the rewritten shuffle in actual a backport of the shuffle from MR2 .
> We mainly ad
Hi: Todd
yes, the rewritten shuffle in actual a backport of the shuffle from MR2 .
We mainly add the following two features:
1) shuffle avoidance
2) work pool
On Fri, Feb 17, 2012 at 3:27 AM, Todd Lipcon wrote:
> Hey Schubert,
>
> Looking at the code on github, it looks like your rewritten shuf
Hey Schubert,
Looking at the code on github, it looks like your rewritten shuffle is
in fact just a backport of the shuffle from MR2. I didn't look closely
- are there any distinguishing factors?
Also, the OOB heartbeat and adaptive heartbeat code seems to be the
same as what's in 1.0?
-Todd
On
Here is the presentation to describe our job,
http://www.slideshare.net/hanborq/hanborq-optimizations-on-hadoop-mapreduce-20120216aWellcome
to give your advises.
It's just a little step, and we are continue to do more improvements,
thanks for your help.
On Thu, Feb 16, 2012 at 11:01 PM, Anty wr
