Remember that the you need to souce your .bashrc For your PATH to be set up.
From: luohui20...@sina.com [mailto:luohui20...@sina.com] Sent: 20 July 2016 11:01 To: user <user@spark.apache.org> Subject: run spark apps in linux crontab hi guys: I add a spark-submit job into my Linux crontab list by the means below ,however none of them works. If I change it to a normal shell script, it is ok. I don't quite understand why. I checked the 8080 web ui of my spark cluster, no job submitted, and there is not messages in /home/hadoop/log. Any idea is welcome. [hadoop@master ~]$ crontab -e 1. 22 21 * * * sh /home/hadoop/shellscripts/run4.sh > /home/hadoop/log and in run4.sh,it wrote: $SPARK_HOME/bin/spark-submit --class com.abc.myclass --total-executor-cores 10 --jars $SPARK_HOME/lib/MyDep.jar $SPARK_HOME/MyJar.jar > /home/hadoop/log 2. 22 21 * * * $SPARK_HOME/bin/spark-submit --class com.abc.myclass --total-executor-cores 10 --jars $SPARK_HOME/lib/MyDep.jar $SPARK_HOME/MyJar.jar > /home/hadoop/log 3. 22 21 * * * /usr/lib/spark/bin/spark-submit --class com.abc.myclass --total-executor-cores 10 --jars /usr/lib/spark/lib/MyDep.jar /usr/lib/spark/MyJar.jar > /home/hadoop/log 4. 22 21 * * * hadoop /usr/lib/spark/bin/spark-submit --class com.abc.myclass --total-executor-cores 10 --jars /usr/lib/spark/lib/MyDep.jar /usr/lib/spark/MyJar.jar > /home/hadoop/log -------------------------------- Thanks&Best regards! San.Luo This email is confidential and may be subject to privilege. If you are not the intended recipient, please do not copy or disclose its content but contact the sender immediately upon receipt.
