Can you paste the stderr from the worker logs? (Found in work/ app-20140625133031-0002/ directory)
Most likely you might need to set SPARK_MASTER_IP in your spark-env.sh file (Not sure why i'm seeing akka.tcp://spark@localhost:56569 instead of akka.tcp://spark@*serverip*:56569) Thanks Best Regards On Thu, Jun 26, 2014 at 2:26 AM, Sameer Tilak <ssti...@live.com> wrote: > Hi All, > > I see the following error messages on my worker nodes. Are they due to > improper cleanup or wrong configuration? Any help with this would be great! > > 14/06/25 12:30:55 INFO SecurityManager: Using Spark's default log4j > profile: org/apache/spark/log4j-defaults.properties > 14/06/25 12:30:55 INFO SecurityManager: Changing view acls to: userid > 14/06/25 12:30:55 INFO SecurityManager: SecurityManager: authentication > disabled; ui acls disabled; users with view permissions: Set(p529444) > 14/06/25 12:30:56 INFO Slf4jLogger: Slf4jLogger started > 14/06/25 12:30:56 INFO Remoting: Starting remoting > 14/06/25 12:30:56 INFO Remoting: Remoting started; listening on addresses > :[akka.tcp://sparkWorker@worker1ip:60276] > 14/06/25 12:30:57 INFO Worker: Starting Spark worker worker1ip:60276 with > 1 cores, 2.7 GB RAM > 14/06/25 12:30:57 INFO Worker: Spark home: > /apps/software/spark-1.0.0-bin-hadoop1 > 14/06/25 12:30:57 INFO WorkerWebUI: Started WorkerWebUI at > http://worker1ip:8081 > 14/06/25 12:30:57 INFO Worker: Connecting to master > spark://serverip:7077... > 14/06/25 12:30:57 INFO Worker: Successfully registered with master > spark://serverip:7077 > 14/06/25 12:32:05 INFO Worker: Asked to launch executor > app-20140625123205-0000/2 for ApproxStrMatch > 14/06/25 12:32:05 INFO ExecutorRunner: Launch command: > "/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.9.x86_64/jre/bin/java" "-cp" > "::/apps/software/spark-1.0.0-bin-hadoop1/conf:/apps/software/spark-1.0.0-bin-hadoop1/lib/spark-assembly-1.0.0-hadoop1.0.4.jar:/apps/hadoop/hadoop-conf" > "-XX:MaxPermSize=128m" "-Xms512M" "-Xmx512M" > "org.apache.spark.executor.CoarseGrainedExecutorBackend" > "akka.tcp://spark@localhost:56569/user/CoarseGrainedScheduler" "2" > "p worker1ip" "1" "akka.tcp://sparkWorker@ worker1ip:60276/user/Worker" > "app-20140625123205-0000" > 14/06/25 12:32:09 INFO Worker: Executor app-20140625123205-0000/2 finished > with state FAILED message Command exited with code 1 exitStatus 1 > 14/06/25 12:32:09 INFO Worker: Asked to launch executor > app-20140625123205-0000/5 for ApproxStrMatch > 14/06/25 12:32:09 INFO ExecutorRunner: Launch command: > "/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.9.x86_64/jre/bin/java" "-cp" > "::/apps/software/spark-1.0.0-bin-hadoop1/conf:/apps/software/spark-1.0.0-bin-hadoop1/lib/spark-assembly-1.0.0-hadoop1.0.4.jar:/apps/hadoop/hadoop-conf" > "-XX:MaxPermSize=128m" "-Xms512M" "-Xmx512M" > "org.apache.spark.executor.CoarseGrainedExecutorBackend" > "akka.tcp://spark@localhost:56569/user/CoarseGrainedScheduler" "5" > "worker1ip" "1" "akka.tcp://sparkWorker@ worker1ip:60276/user/Worker" > "app-20140625123205-0000" > 14/06/25 12:32:12 INFO Worker: Executor app-20140625123205-0000/5 finished > with state FAILED message Command exited with code 1 exitStatus 1 > 14/06/25 12:32:12 INFO Worker: Asked to launch executor > app-20140625123205-0000/9 for ApproxStrMatch > 14/06/25 12:32:12 INFO ExecutorRunner: Launch command: > "/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.9.x86_64/jre/bin/java" "-cp" > "::/apps/software/spark-1.0.0-bin-hadoop1/conf:/apps/software/spark-1.0.0-bin-hadoop1/lib/spark-assembly-1.0.0-hadoop1.0.4.jar:/apps/hadoop/hadoop-conf" > "-XX:MaxPermSize=128m" "-Xms512M" "-Xmx512M" > "org.apache.spark.executor.CoarseGrainedExecutorBackend" > "akka.tcp://spark@localhost:56569/user/CoarseGrainedScheduler" "9" > "worker1ip" "1" "akka.tcp://sparkWorker@ worker1ip:60276/user/Worker" > "app-20140625123205-0000" > 14/06/25 12:32:16 INFO Worker: Asked to kill executor > app-20140625123205-0000/9 > 14/06/25 12:32:16 INFO ExecutorRunner: Runner thread for executor > app-20140625123205-0000/9 interrupted > 14/06/25 12:32:16 INFO ExecutorRunner: Killing process! > 14/06/25 12:32:16 INFO Worker: Executor app-20140625123205-0000/9 finished > with state KILLED > 14/06/25 13:28:44 INFO Worker: Asked to launch executor > app-20140625132844-0001/2 for ApproxStrMatch > 14/06/25 13:28:44 INFO ExecutorRunner: Launch command: > "/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.9.x86_64/jre/bin/java" "-cp" > "::/apps/software/spark-1.0.0-bin-hadoop1/conf:/apps/software/spark-1.0.0-bin-hadoop1/lib/spark-assembly-1.0.0-hadoop1.0.4.jar:/apps/hadoop/hadoop-conf" > "-XX:MaxPermSize=128m" "-Xms512M" "-Xmx512M" > "org.apache.spark.executor.CoarseGrainedExecutorBackend" > "akka.tcp://spark@localhost:46648/user/CoarseGrainedScheduler" "2" > "worker1ip" "1" "akka.tcp://sparkWorker@ worker1ip:60276/user/Worker" > "app-20140625132844-0001" > 14/06/25 13:28:48 INFO Worker: Executor app-20140625132844-0001/2 finished > with state FAILED message Command exited with code 1 exitStatus 1 > 14/06/25 13:28:48 INFO Worker: Asked to launch executor > app-20140625132844-0001/5 for ApproxStrMatch > 14/06/25 13:28:48 INFO ExecutorRunner: Launch command: > "/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.9.x86_64/jre/bin/java" "-cp" > "::/apps/software/spark-1.0.0-bin-hadoop1/conf:/apps/software/spark-1.0.0-bin-hadoop1/lib/spark-assembly-1.0.0-hadoop1.0.4.jar:/apps/hadoop/hadoop-conf" > "-XX:MaxPermSize=128m" "-Xms512M" "-Xmx512M" > "org.apache.spark.executor.CoarseGrainedExecutorBackend" > "akka.tcp://spark@localhost:46648/user/CoarseGrainedScheduler" "5" > "worker1ip" "1" "akka.tcp://sparkWorker@ worker1ip:60276/user/Worker" > "app-20140625132844-0001" > 14/06/25 13:28:51 INFO Worker: Executor app-20140625132844-0001/5 finished > with state FAILED message Command exited with code 1 exitStatus 1 > 14/06/25 13:28:51 INFO Worker: Asked to launch executor > app-20140625132844-0001/8 for ApproxStrMatch > 14/06/25 13:28:51 INFO ExecutorRunner: Launch command: > "/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.9.x86_64/jre/bin/java" "-cp" > "::/apps/software/spark-1.0.0-bin-hadoop1/conf:/apps/software/spark-1.0.0-bin-hadoop1/lib/spark-assembly-1.0.0-hadoop1.0.4.jar:/apps/hadoop/hadoop-conf" > "-XX:MaxPermSize=128m" "-Xms512M" "-Xmx512M" > "org.apache.spark.executor.CoarseGrainedExecutorBackend" > "akka.tcp://spark@localhost:46648/user/CoarseGrainedScheduler" "8" > "worker1ip" "1" "akka.tcp://sparkWorker@ worker1ip:60276/user/Worker" > "app-20140625132844-0001" > 14/06/25 13:28:54 INFO Worker: Executor app-20140625132844-0001/8 finished > with state FAILED message Command exited with code 1 exitStatus 1 > 14/06/25 13:30:31 INFO Worker: Asked to launch executor > app-20140625133031-0002/2 for ApproxStrMatch > 14/06/25 13:30:31 INFO ExecutorRunner: Launch command: > "/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.9.x86_64/jre/bin/java" "-cp" > "::/apps/software/spark-1.0.0-bin-hadoop1/conf:/apps/software/spark-1.0.0-bin-hadoop1/lib/spark-assembly-1.0.0-hadoop1.0.4.jar:/apps/hadoop/hadoop-conf" > "-XX:MaxPermSize=128m" "-Xms512M" "-Xmx512M" > "org.apache.spark.executor.CoarseGrainedExecutorBackend" > "akka.tcp://spark@localhost:42235/user/CoarseGrainedScheduler" "2" > "worker1ip" "1" "akka.tcp://sparkWorker@ worker1ip:60276/user/Worker" > "app-20140625133031-0002" > 14/06/25 13:30:34 INFO Worker: Executor app-20140625133031-0002/2 finished > with state FAILED message Command exited with code 1 exitStatus 1 > 14/06/25 13:30:34 INFO Worker: Asked to launch executor > app-20140625133031-0002/5 for ApproxStrMatch > 14/06/25 13:30:35 INFO ExecutorRunner: Launch command: > "/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.9.x86_64/jre/bin/java" "-cp" > "::/apps/software/spark-1.0.0-bin-hadoop1/conf:/apps/software/spark-1.0.0-bin-hadoop1/lib/spark-assembly-1.0.0-hadoop1.0.4.jar:/apps/hadoop/hadoop-conf" > "-XX:MaxPermSize=128m" "-Xms512M" "-Xmx512M" > "org.apache.spark.executor.CoarseGrainedExecutorBackend" > "akka.tcp://spark@localhost:42235/user/CoarseGrainedScheduler" "5" > "worker1ip" "1" "akka.tcp://sparkWorker@worker1ip:60276/user/Worker" > "app-20140625133031-0002" > 14/06/25 13:30:36 INFO Worker: Asked to kill executor > app-20140625133031-0002/5 > 14/06/25 13:30:36 INFO Worker: Executor app-20140625133031-0002/5 finished > with state KILLED > 14/06/25 13:30:36 INFO ExecutorRunner: Runner thread for executor > app-20140625133031-0002/5 interrupted > 14/06/25 13:30:36 INFO ExecutorRunner: Killing process! >
