I am not manually setting spark.mesos.coarse to true neither in the code
nor in any configuration file. So it should be taking the default value and
so it should be running in fine-grained mode.
When i try to log, conf.get("spark.mesos.coarse"), my application exited
with this error:
Exception in thread "main" java.util.NoSuchElementException:
spark.mesos.coarse
at
org.apache.spark.SparkConf$$anonfun$get$1.apply(SparkConf.scala:169)
at
org.apache.spark.SparkConf$$anonfun$get$1.apply(SparkConf.scala:169)
at scala.Option.getOrElse(Option.scala:120)
at org.apache.spark.SparkConf.get(SparkConf.scala:169)
On Wed, Sep 2, 2015 at 4:30 PM, Akhil Das <ak...@sigmoidanalytics.com>
wrote:
>
> "Initial job has not accepted any resources; check your cluster UI to
> ensure that workers are registered and have sufficient resources".
>
> I'm assuming you are submitting the job in coarse-grained mode, in that
> case make sure you are asking for the available resources.
>
> If you want to submit multiple applications and run them side-by-side then
> you can submit the application in fine-grained mode.
>
> Read more over here
> http://spark.apache.org/docs/latest/running-on-mesos.html#mesos-run-modes
>
> Thanks
> Best Regards
>
> On Wed, Sep 2, 2015 at 4:02 PM, srungarapu vamsi <srungarapu1...@gmail.com
> > wrote:
>
>> Hi,
>>
>> I am using a mesos cluster to run my spark jobs.
>> I have one mesos-master and two mesos-slaves setup on 2 machines.
>> On one machine, master and slave are setup and on the second machine
>> mesos-slave is setup
>> I run these on m3-large ec2 instances.
>>
>> 1. When i try to submit two jobs using spark-submit in parallel, one job
>> hangs with the message : "Initial job has not accepted any resources; check
>> your cluster UI to ensure that workers are registered and have sufficient
>> resources". But when i check on the mesos cluster UI which runs at 5050
>> port, i can see idle memory which can be used by the hanging job. But
>> number of idle cores is 1.
>> So, does this mean that cores are pinned to spark-submit and no other
>> spark-submit can get the core till the running spark-submit completes ?
>>
>> 2. Assumption : "submitting multiple spark-jobs using spark-submit has
>> the above mentioned problem ".
>> Now my task is to run a spark-streaming job which reads from kafka and
>> does some precomputation.
>> The nature of my pre-computation jobs are in such a way that, each
>> pre-compute jobs has few mutually exclusive tasks to complete where all the
>> tasks have inherent tree structure in them. i.e A task initiates few other
>> tasks and they initiate further more tasks.
>> I already have spark jobs which run as a batch job to perform the
>> pre-computations mentioned above. Now, is it a good idea to convert these
>> precompuations jobs into akka actors ?
>>
>> 3. If at all running multiple spark-submit jobs with shared CPU is
>> possible, for the scenario explained in Point.2, which approach is better :
>> "precomputation jobs as actors" vs "multiple spark-submits" ?
>>
>> Any pointers to clear my above doubts is highly appreciated.
>> --
>> /Vamsi
>>
>
>
--
/Vamsi
