Perfect, thank you Harsh, method that I was looking for was markUnresponsiveTasks..
So in general if I sum up my understanding of fairscheduling is as follow: - Pools assignments and ordering is done at tasktrackers requests and is based primarily on how far is the pool bellow its minshare, afterwards on runningTasks/weight. Fairness is provided by the preemption mechanism which occurs every 15sec and checks the exact fair share that should be allocated to every pool (Fairscheduler#preemptTasksIfNecessary()). - After the pool a job is chosen based on the internal pool scheduling policy (fair or fifo-by-arriving-time) - Task assignment from a chosen job is based on task's data and the tasktracker that is asking for a new task. Task assignments cannot be manipulated by the schedulers and the assignment procedure is the same for all schedulers (first try to assign non running tasks, then local and at last speculative tasks). Type of task locality is defined by delay scheduling algorithm. - Preemption mechanism for slow task is done on the tasktracker (TaskTracker#markUnresponsiveTasks). Other thing that I am just wondering is why fairsharing calculation (SchedulingAlgorithms#computeFairShare) is not taken into account in the sorting of pools at every tasktracker request? If I am not wrong the time complexity of finding a right slotToWeight ratio R is O(log(R)*jobs). Is maybe this the reason for not including it in every tasktracker request? On 9 June 2012 15:38, Harsh J <ha...@cloudera.com> wrote: > Merto, > > You seem to be looking for the speculative execution code? > > For MR1, in 1.x or 0.20.x and below, you can find that in the > JobInProgress#findNewMapTask method (go down to the bit which does > speculative execution logic for maps and reduces depending on if the > job enabled it for itself). > > For MR2, in 2.x and above, since it offers a way to have pluggable > speculative algorithms, the default implementation (a slightly > advanced version of the one in 1.0) is available in the > DefaultSpeculator class (under the mapreduce-client-app package). This > is used by the MRAppMaster (and is event driven). > > If you are instead asking about the task timeout monitor (that kills > tasks that haven't reported for a period): > > For MR1 it is in TaskTracker#markUnresponsiveTasks method (that uses > timeout values set per task (from a job)). > > For MR2 similar code is in TaskHeartbeatHandler#PingChecker thread > class. This is again in the mapreduce-client-app package. > > Preemption is a different operation than monitoring slow jobs/tasks. > > I believe you should be able to, with any decent IDE, follow the above > entry points and then be able to drill up/down from there to analyze > the code. In case you get stuck somewhere in understanding something > around these features, feel free to ask us here! > > On Fri, Jun 8, 2012 at 9:14 PM, Merto Mertek <masmer...@gmail.com> wrote: > > I am interested in which component of hadoop and how is managed slow task > > execution. > > > > Is this a tasktracker, jobtracker or a scheduler? > > > > In fairscheduling (0.20.205) there exist a preemption mechanism but it > just > > handles min and fairshares and kills last submited tips.. > > > > Thanks. > > > > -- > Harsh J >
