Re: Setting Spark Worker Memory

run JPS like below jps 19724 SparkSubmit 10612 Worker and do ps awx|grep PID for each number that represents these two descriptions. something like ps awx|grep 30208 30208 pts/2Sl+1:05 /usr/java/latest/bin/java -cp /home/hduser/jars/jconn4.jar:/home/hduser/jars/ojdbc6.jar:/usr/lib/sp

Re: Setting Spark Worker Memory

yes, i m running this as standalone mode. On Wed, May 11, 2016 at 6:23 PM, Mich Talebzadeh wrote: > are you running this in standalone mode? that is one physical host, and > the executor will live inside the driver. > > > > Dr Mich Talebzadeh > > > > LinkedIn * > https://www.linkedin.com/profi

Re: Setting Spark Worker Memory

are you running this in standalone mode? that is one physical host, and the executor will live inside the driver. Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw

Re: Setting Spark Worker Memory

yes, On Wed, May 11, 2016 at 5:43 PM, Deepak Sharma wrote: > Since you are registering workers from the same node , do you have enough > cores and RAM(In this case >=9 cores and > = 24 GB ) on this > node(11.14.224.24)? > > Thanks > Deepak > > On Wed, May 11, 2016 at 9:08 PM, شجاع الرحمن بیگ >

Re: Setting Spark Worker Memory

Since you are registering workers from the same node , do you have enough cores and RAM(In this case >=9 cores and > = 24 GB ) on this node(11.14.224.24)? Thanks Deepak On Wed, May 11, 2016 at 9:08 PM, شجاع الرحمن بیگ wrote: > Hi All, > > I need to set same memory and core for each worker on sa

Setting Spark Worker Memory

Hi All, I need to set same memory and core for each worker on same machine and for this purpose, I have set the following properties in conf/spark-env.sh export SPARK_EXECUTOR_INSTANCE=3 export SPARK_WORKER_CORES=3 export SPARK_WORKER_MEMORY=8g but only one worker is getting desired memory and c