Is this a standalone set up single host where executor runs inside the
driver?
also run
*free -t*
To see the virtual memory usage which is basically swap space
free -t
total used free shared buffers cached
Mem: 24546308 24268760 277548 0 1088236 15168668
-/+ buffers/cache: 8011856 16534452
Swap: 2031608 304 2031304
Total: 26577916 24269064 2308852
Dr Mich Talebzadeh
LinkedIn *
https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
http://talebzadehmich.wordpress.com
On 13 May 2016 at 07:36, Jone Zhang <[email protected]> wrote:
> mich, Do you want this
>
> ==========================================================================================
> [running][email protected]:/data/home/hive/conf$ ps aux | grep SparkPi
> mqq 20070 3.6 0.8 10445048 267028 pts/16 Sl+ 13:09 0:11
> /data/home/jdk/bin/java
> -Dlog4j.configuration=file:///data/home/spark/conf/log4j.properties
> -cp
> /data/home/spark/lib/*:/data/home/hadoop/share/hadoop/common/*:/data/home/hadoop/share/hadoop/common/lib/*:/data/home/hadoop/share/hadoop/yarn/*:/data/home/hadoop/share/hadoop/yarn/lib/*:/data/home/hadoop/share/hadoop/hdfs/*:/data/home/hadoop/share/hadoop/hdfs/lib/*:/data/home/hadoop/share/hadoop/tools/*:/data/home/hadoop/share/hadoop/mapreduce/*:/data/home/spark/conf/:/data/home/spark/lib/spark-assembly-1.4.1-hadoop2.5.1_150903.jar:/data/home/spark/lib/datanucleus-api-jdo-3.2.6.jar:/data/home/spark/lib/datanucleus-core-3.2.10.jar:/data/home/spark/lib/datanucleus-rdbms-3.2.9.jar:/data/home/hadoop/conf/:/data/home/hadoop/conf/:/data/home/spark/lib/*:/data/home/hadoop/share/hadoop/common/*:/data/home/hadoop/share/hadoop/common/lib/*:/data/home/hadoop/share/hadoop/yarn/*:/data/home/hadoop/share/hadoop/yarn/lib/*:/data/home/hadoop/share/hadoop/hdfs/*:/data/home/hadoop/share/hadoop/hdfs/lib/*:/data/home/hadoop/share/hadoop/tools/*:/data/home/hadoop/share/hadoop/mapreduce/*
> -XX:MaxPermSize=256m org.apache.spark.deploy.SparkSubmit --master
> yarn-cluster --class org.apache.spark.examples.SparkPi --queue spark
> --num-executors 4
> /data/home/spark/lib/spark-examples-1.4.1-hadoop2.5.1.jar 100000000
> mqq 22410 0.0 0.0 110600 1004 pts/8 S+ 13:14 0:00 grep
> SparkPi
> [running][email protected]:/data/home/hive/conf$ top -p 20070
>
> top - 13:14:48 up 504 days, 19:17, 19 users, load average: 1.41, 1.10,
> 0.99
> Tasks: 1 total, 0 running, 1 sleeping, 0 stopped, 0 zombie
> Cpu(s): 18.1%us, 2.7%sy, 0.0%ni, 74.4%id, 4.5%wa, 0.0%hi, 0.2%si,
> 0.0%st
> Mem: 32740732k total, 31606288k used, 1134444k free, 475908k buffers
> Swap: 2088952k total, 61076k used, 2027876k free, 27594452k cached
>
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 20070 mqq 20 0 10.0g 260m 32m S 0.0 0.8 0:11.38 java
>
> ==========================================================================================
>
> Harsh, physical cpu cores is 1, virtual cpu cores is 4
>
> Thanks.
>
> 2016-05-13 13:08 GMT+08:00, Harsh J <[email protected]>:
> > How many CPU cores are on that machine? Read http://qr.ae/8Uv3Xq
> >
> > You can also confirm the above by running the pmap utility on your
> process
> > and most of the virtual memory would be under 'anon'.
> >
> > On Fri, 13 May 2016 09:11 jone, <[email protected]> wrote:
> >
> >> The virtual memory is 9G When i run org.apache.spark.examples.SparkPi
> >> under yarn-cluster model,which using default configurations.
> >> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+
> >> COMMAND
> >>
> >> 4519 mqq 20 0 9041 <2009041>m 248m 26m S 0.3 0.8 0:19.85
> >> java
> >> I am curious why is so high?
> >>
> >> Thanks.
> >>
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [email protected]
> For additional commands, e-mail: [email protected]
>
>
