Re:run spark apps in linux crontab

[focus]

Hi, I just meet this problem, too! The reason is crontab runtime doesn't have 
the variables you defined, such as $SPARK_HOME.
I defined the $SPARK_HOME and other variables in /etc/profile like this:


export $MYSCRIPTS=/opt/myscripts
export $SPARK_HOME=/opt/spark


then, in my crontab job script daily_job.sh


#!/bin/sh


. /etc/profile


$SPARK_HOME/bin/spark-submit $MYSCRIPTS/fix_fh_yesterday.py


then, in crontab -e


0 8 * * * /home/user/daily_job.sh


hope this helps~








------------------ Original ------------------
From: "luohui20001"<luohui20...@sina.com>; 
Date: 2016??7??20??(??????) ????6:00
To: "user@spark.apache.org"<user@spark.apache.org>; 
Subject: run spark apps in linux crontab



hi guys:
      I add a spark-submit job into my Linux crontab list by the means below 
,however none of them works. If I change it to a normal shell script, it is ok. 
I don't quite understand why. I checked the 8080 web ui of my spark cluster, no 
job submitted, and there is not messages in /home/hadoop/log. 
      Any idea is welcome.


[hadoop@master ~]$ crontab -e
1.
22 21 * * * sh /home/hadoop/shellscripts/run4.sh > /home/hadoop/log 


and in run4.sh,it wrote:
$SPARK_HOME/bin/spark-submit --class com.abc.myclass --total-executor-cores 10 
--jars $SPARK_HOME/lib/MyDep.jar $SPARK_HOME/MyJar.jar  > /home/hadoop/log 


2.
22 21 * * * $SPARK_HOME/bin/spark-submit --class com.abc.myclass 
--total-executor-cores 10 --jars $SPARK_HOME/lib/MyDep.jar 
$SPARK_HOME/MyJar.jar  > /home/hadoop/log 


3.
22 21 * * * /usr/lib/spark/bin/spark-submit --class com.abc.myclass 
--total-executor-cores 10 --jars /usr/lib/spark/lib/MyDep.jar 
/usr/lib/spark/MyJar.jar  > /home/hadoop/log 


4.
22 21 * * * hadoop /usr/lib/spark/bin/spark-submit --class com.abc.myclass 
--total-executor-cores 10 --jars /usr/lib/spark/lib/MyDep.jar 
/usr/lib/spark/MyJar.jar  > /home/hadoop/log 


--------------------------------

  

 Thanks&amp;Best regards!
San.Luo