On Thu, Feb 16, 2012 at 8:25 PM, Schubert Zhang <zson...@gmail.com> wrote: > 1) it should be sort-avoidance.
right - that's a nice improvement, looking forward to getting that in trunk at some point. > 2) work pool (like Tenzing) > Looking at the code, it seems you only support the default task executor. Do you have plans to support run-as-user through the linux task-controller? It's a requirement for secure environments. But, it makes the worker pool model a little tougher since you can't share a JVM cross-user. Also, how does class-unloading and reloading interact with this model? > Sorry ,the adaptive heartbeat code is not in this github code, we are > discussing it. > > > > On Fri, Feb 17, 2012 at 11:00 AM, Anty <anty....@gmail.com> wrote: >> >> Hi: Todd >> >> yes, the rewritten shuffle in actual a backport of the shuffle from MR2 . >> We mainly add the following two features: >> 1) shuffle avoidance >> 2) work pool >> >> >> On Fri, Feb 17, 2012 at 3:27 AM, Todd Lipcon <t...@cloudera.com> wrote: >>> >>> Hey Schubert, >>> >>> Looking at the code on github, it looks like your rewritten shuffle is >>> in fact just a backport of the shuffle from MR2. I didn't look closely >>> - are there any distinguishing factors? >>> Also, the OOB heartbeat and adaptive heartbeat code seems to be the >>> same as what's in 1.0? >>> >>> -Todd >>> >>> On Thu, Feb 16, 2012 at 9:44 AM, Schubert Zhang <zson...@gmail.com> >>> wrote: >>> > Here is the presentation to describe our job, >>> > >>> > http://www.slideshare.net/hanborq/hanborq-optimizations-on-hadoop-mapreduce-20120216a >>> > Wellcome to give your advises. >>> > It's just a little step, and we are continue to do more improvements, >>> > thanks >>> > for your help. >>> > >>> > >>> > >>> > >>> > On Thu, Feb 16, 2012 at 11:01 PM, Anty <anty....@gmail.com> wrote: >>> >> >>> >> Hi: Guys >>> >> We just deliver a optimized hadoop , if you are interested, Pls >>> >> refer to https://github.com/hanborq/hadoop >>> >> >>> >> -- >>> >> Best Regards >>> >> Anty Rao >>> > >>> > >>> >>> >>> >>> -- >>> Todd Lipcon >>> Software Engineer, Cloudera >> >> >> >> >> -- >> Best Regards >> Anty Rao > > -- Todd Lipcon Software Engineer, Cloudera
