You are right, Hamel. It should get 10 TB /2. And In hadoop-2.7.0, it is working fine. But in hadoop-2.5.1, it gets only 10TB/230. The same configuration used in both versions. So i think a JIRA could have fixed the issue after hadoop-2.5.1.
On Thu, Feb 25, 2016 at 1:28 AM, Hamel Kothari <hamelkoth...@gmail.com> wrote: > The instantaneous fair share is what Queue B should get according to the > code (and my experience). Assuming your queues are all equal it would be > 10TB/2. > > I can't help much more unless I can see your config files and ideally also > the YARN Scheduler UI to get an idea of what your queues/actual resource > usage is like. Logs from each of your Spark applications would also be > useful. Basically the more info the better. > > On Wed, Feb 24, 2016 at 2:52 PM Prabhu Joseph <prabhujose.ga...@gmail.com> > wrote: > >> Hi Hamel, >> >> Thanks for looking into the issue. What i am not understanding is, >> after preemption what is the share that the second queue gets in case if >> the first queue holds the entire cluster resource without releasing, is it >> instantaneous fair share or fair share. >> >> Queue A and B are there (total 230 queues), total cluster resource >> is 10TB, 3000 cores. If a job submitted into queue A, it will get 10TB, >> 3000 cores and it is not releasing any resource. Now if a second job >> submitted into queue B, so preemption definitely will happen, but what is >> the share queue B will get after preemption. *Is it <10 TB , 3000> / 2 >> or <10TB,3000> / 230* >> >> We find, after preemption queue B gets only <10TB,3000> / 230, because >> the first job is holding the resource. In case if first job releases the >> resource, the second queue will get <10TB,3000> /2 based on higher priority >> and reservation. >> >> The question is how much preemption tries to preempt the queue A if it >> holds the entire resource without releasing? Could not able to share the >> actual configuration, but the answer to the question here will help us. >> >> >> Thanks, >> Prabhu Joseph >> >> >> >> >> On Wed, Feb 24, 2016 at 10:03 PM, Hamel Kothari <hamelkoth...@gmail.com> >> wrote: >> >>> If all queues are identical, this behavior should not be happening. >>> Preemption as designed in fair scheduler (IIRC) takes place based on the >>> instantaneous fair share, not the steady state fair share. The fair >>> scheduler docs >>> <https://hadoop.apache.org/docs/r2.7.1/hadoop-yarn/hadoop-yarn-site/FairScheduler.html> >>> aren't super helpful on this but it does say in the Monitoring section that >>> preemption won't take place if you're less than your instantaneous fair >>> share (which might imply that it would occur if you were over your inst. >>> fair share and someone had requested resources). The code for >>> FairScheduler.resToPreempt >>> <http://grepcode.com/file/repo1.maven.org/maven2/org.apache.hadoop/hadoop-yarn-server-resourcemanager/2.7.1/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java#FairScheduler.resToPreempt%28org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue%2Clong%29> >>> also seems to use getFairShare rather than getSteadyFairShare() for >>> preemption so that would imply that it is using instantaneous fair share >>> rather than steady state. >>> >>> Could you share your YARN site/fair-scheduler and Spark configurations? >>> Could you also share the YARN Scheduler UI (specifically the top of of the >>> RM which shows how many resources are in use)? >>> >>> Since it's not likely due to steady state fair share, some other >>> possible reasons why this might be taking place (this is not remotely >>> conclusive but with no information this is what comes to mind): >>> - You're not reaching >>> yarn.scheduler.fair.preemption.cluster-utilization-threshold. Perhaps >>> due to core/memory ratio inconsistency with the cluster. >>> - Your second job doesn't have a sufficient level of parallelism to >>> request more executors than what it is recieving (perhaps there are fewer >>> than 13 tasks at any point in time) and you don't have >>> spark.dynamicAllocation.minExecutors set? >>> >>> -Hamel >>> >>> On Tue, Feb 23, 2016 at 8:20 PM Prabhu Joseph < >>> prabhujose.ga...@gmail.com> wrote: >>> >>>> Hi All, >>>> >>>> A YARN cluster with 352 Nodes (10TB, 3000cores) and has Fair Scheduler >>>> with root queue having 230 queues. >>>> >>>> Each Queue is configured with maxResources equal to Total Cluster >>>> Resource. When a Spark job is submitted into a queue A, it is given with >>>> 10TB, 3000 cores according to instantaneous Fair Share and it is holding >>>> the entire resource without releasing. After some time, when another job is >>>> submitted into other queue B, it will get the Fair Share 45GB and 13 cores >>>> i.e (10TB,3000 cores)/230 using Preemption. Now if some more jobs are >>>> submitted into queue B, all the jobs in B has to share the 45GB and 13 >>>> cores. Whereas the job which is in queue A holds the entire cluster >>>> resource affecting the other jobs. >>>> This kind of issue often happens when a Spark job submitted first >>>> which holds the entire cluster resource. What is the best way to fix this >>>> issue. Can we make preemption to happen for instantaneous fair share >>>> instead of fair share, will it help. >>>> >>>> Note: >>>> >>>> 1. We do not want to give weight for particular queue. Because all the >>>> 240 queues are critical. >>>> 2. Changing the queues into nested does not solve the issue. >>>> 3. Adding maxResource to queue won't allow the first job to pick >>>> entire cluster resource, but still configuring the optimal maxResource for >>>> 230 queue is difficult and also the first job can't use the entire cluster >>>> resource when the cluster is idle. >>>> 4. We do not want to handle it in Spark ApplicationMaster, then we need >>>> to check for other new YARN application type with similar behavior. We want >>>> YARN to control this behavior by killing the resources which is hold by >>>> first job for longer period. >>>> >>>> >>>> Thanks, >>>> Prabhu Joseph >>>> >>>> >>
