+1 I agree with Steve that sometimes you need to redirect where you want the work to occur.
Over time, your cloud will not have homogenous data nodes. You may end up with a cluster of nodes that have a Fermi card (NVIDA CUDA enabled cards) where you want to do some serious number crunching. [ I don't believe if CUDA supports Java, but you get the idea...] So in theory, you will want to dictate where the work will be performed. -Mike PS. Sorry if this is a bad example. I was talking with my Uncle over the weekend and he's in to some serious number crunching. -----Original Message----- From: Steve Loughran [mailto:ste...@apache.org] Sent: Monday, May 17, 2010 6:47 AM To: common-dev@hadoop.apache.org Subject: Re: Task scheduler Saurabh Agarwal wrote: > Hemanth, > > > Thanks!! > Saurabh Agarwal > > > On Fri, May 14, 2010 at 9:49 AM, Hemanth Yamijala <yhema...@gmail.com>wrote: > >> Saurabh, >> >>> let me re frame my question I wanted to knowhow job tracker decides the >>> assignment of input splits to task tracker based on task tracker's data >>> locality. Where is this policy defined? Is it pluggable? >> Sorry, I misunderstood your question then. This code is in >> o.a.h.mapred.JobInProgress. It is likely spread across many methods in >> the class. But a good starting point could be from methods like >> obtainNewMapTask or obtainNewReduceTask. >> >> At the moment, this policy is not pluggable. But I know there have >> been discussions (possibly even a JIRA, though I can't locate any now) >> asking for this capability. >> +1 to having some plugin interface in 0.22+ to give you control. My fomer colleague russ perry did some rendering with Hadoop where he wanted the work done not where the input data was, but where the output data was needed; there was no way to do this http://www.hpl.hp.com/techreports/2009/HPL-2009-345.pdf The information contained in this communication may be CONFIDENTIAL and is intended only for the use of the recipient(s) named above. If you are not the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication, or any of its contents, is strictly prohibited. If you have received this communication in error, please notify the sender and delete/destroy the original message and any copy of it from your computer or paper files.
