Hi Saisai, Would you mind giving me some tips about this problem? After check YARN RM logs, I think Spark application didn't request resources from it, So, I guess this problem is none of YARN's business. and the spark conf of my cluster will be list in the following:
spark.shuffle.service.enabled true spark.dynamicAllocation.enabled true spark.shuffle.service.port 7337 spark.dynamicAllocation.maxExecutors 50 spark.dynamicAllocation.minExecutors 50 If I change spark.dynamicAllocation.minExecutors from 50 to 49 or else less than 50 it will work. So, I think even if it isn't a bug about Dynamic Alloction, something else may be wrong. And, you said that you are OK with min and max executors to the same number, Could you tell me your test cluster environment? Thanks 2015-11-24 13:10 GMT+08:00 Saisai Shao <[email protected]>: > I don't think it is a bug, maybe something wrong with your Spark / Yarn > configurations. > > On Tue, Nov 24, 2015 at 12:13 PM, 谢廷稳 <[email protected]> wrote: > >> OK,the YARN cluster was used by myself,it have 6 node witch can run over >> 100 executor, and the YARN RM logs showed that the Spark application did >> not requested resource from it. >> >> Is this a bug? Should I create a JIRA for this problem? >> >> 2015-11-24 12:00 GMT+08:00 Saisai Shao <[email protected]>: >> >>> OK, so this looks like your Yarn cluster does not allocate containers >>> which you supposed should be 50. Does the yarn cluster have enough resource >>> after allocating AM container, if not, that is the problem. >>> >>> The problem not lies in dynamic allocation from my guess of your >>> description. I said I'm OK with min and max executors to the same number. >>> >>> On Tue, Nov 24, 2015 at 11:54 AM, 谢廷稳 <[email protected]> wrote: >>> >>>> Hi Saisai, >>>> I'm sorry for did not describe it clearly,YARN debug log said I have 50 >>>> executors,but ResourceManager showed that I only have 1 container for the >>>> AppMaster. >>>> >>>> I have checked YARN RM logs,after AppMaster changed state from ACCEPTED >>>> to RUNNING,it did not have log about this job any more.So,the problem is I >>>> did not have any executor but ExecutorAllocationManager think I have.Would >>>> you minding having a test in your cluster environment? >>>> Thanks, >>>> Weber >>>> >>>> 2015-11-24 11:00 GMT+08:00 Saisai Shao <[email protected]>: >>>> >>>>> I think this behavior is expected, since you already have 50 executors >>>>> launched, so no need to acquire additional executors. You change is not >>>>> solid, it is just hiding the log. >>>>> >>>>> Again I think you should check the logs of Yarn and Spark to see if >>>>> executors are started correctly. Why resource is still not enough where >>>>> you >>>>> already have 50 executors. >>>>> >>>>> On Tue, Nov 24, 2015 at 10:48 AM, 谢廷稳 <[email protected]> wrote: >>>>> >>>>>> Hi SaiSai, >>>>>> I have changed "if (numExecutorsTarget >= maxNumExecutors)" to "if >>>>>> (numExecutorsTarget > maxNumExecutors)" of the first line in the >>>>>> ExecutorAllocationManager#addExecutors() and it rans well. >>>>>> In my opinion,when I was set minExecutors equals maxExecutors,when >>>>>> the first time to add Executors,numExecutorsTarget equals maxNumExecutors >>>>>> and it repeat printe "DEBUG ExecutorAllocationManager: Not adding >>>>>> executors because our current target total is already 50 (limit 50)". >>>>>> Thanks >>>>>> Weber >>>>>> >>>>>> 2015-11-23 21:00 GMT+08:00 Saisai Shao <[email protected]>: >>>>>> >>>>>>> Hi Tingwen, >>>>>>> >>>>>>> Would you minding sharing your changes in >>>>>>> ExecutorAllocationManager#addExecutors(). >>>>>>> >>>>>>> From my understanding and test, dynamic allocation can be worked >>>>>>> when you set the min to max number of executors to the same number. >>>>>>> >>>>>>> Please check your Spark and Yarn log to make sure the executors are >>>>>>> correctly started, the warning log means currently resource is not >>>>>>> enough >>>>>>> to submit tasks. >>>>>>> >>>>>>> Thanks >>>>>>> Saisai >>>>>>> >>>>>>> >>>>>>> On Mon, Nov 23, 2015 at 8:41 PM, 谢廷稳 <[email protected]> wrote: >>>>>>> >>>>>>>> Hi all, >>>>>>>> I ran a SparkPi on YARN with Dynamic Allocation enabled and set >>>>>>>> spark.dynamicAllocation.maxExecutors >>>>>>>> equals >>>>>>>> spark.dynamicAllocation.minExecutors,then I submit an application >>>>>>>> using: >>>>>>>> ./bin/spark-submit --class org.apache.spark.examples.SparkPi >>>>>>>> --master yarn-cluster --driver-memory 4g --executor-memory 8g >>>>>>>> lib/spark-examples*.jar 200 >>>>>>>> >>>>>>>> then, this application was submitted successfully, but the >>>>>>>> AppMaster always saying “15/11/23 20:13:08 WARN >>>>>>>> cluster.YarnClusterScheduler: Initial job has not accepted any >>>>>>>> resources; >>>>>>>> check your cluster UI to ensure that workers are registered and have >>>>>>>> sufficient resources” >>>>>>>> and when I open DEBUG,I found “15/11/23 20:24:00 DEBUG >>>>>>>> ExecutorAllocationManager: Not adding executors because our current >>>>>>>> target >>>>>>>> total is already 50 (limit 50)” in the console. >>>>>>>> >>>>>>>> I have fixed it by modifying code in >>>>>>>> ExecutorAllocationManager.addExecutors,Does this a bug or it was >>>>>>>> designed >>>>>>>> that we can’t set maxExecutors equals minExecutors? >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Weber >>>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> >
