Hi spark users:
After I submitted a SparkPi job to spark, the driver crashed at the end of the
job with the following log:
WARN EventLoggingListener: Event log dir
file:/d:/data/SparkWorker/work/driver-20150607200517-0002/logs/event does not
exists, will newly create one.
Exception in thread "main" java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.spark.deploy.worker.DriverWrapper$.main(DriverWrapper.scala:59)
at
org.apache.spark.deploy.worker.DriverWrapper.main(DriverWrapper.scala)
Caused by: java.lang.NullPointerException
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1010)
at org.apache.hadoop.util.Shell.runCommand(Shell.java:445)
at org.apache.hadoop.util.Shell.run(Shell.java:418)
at
org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:650)
at org.apache.hadoop.util.Shell.execCommand(Shell.java:739)
at org.apache.hadoop.util.Shell.execCommand(Shell.java:722)
at
org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:633)
at
org.apache.hadoop.fs.FilterFileSystem.setPermission(FilterFileSystem.java:467)
at
org.apache.spark.scheduler.EventLoggingListener.start(EventLoggingListener.scala:135)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:401)
at org.apache.spark.examples.SparkPi$.main(SparkPi.scala:28)
at org.apache.spark.examples.SparkPi.main(SparkPi.scala)
>From the log, I can see that the driver has added jars from HDFS, connected to
>master, scheduled executors and all the executors were running. And then this
>error occurred.
The command I use to submit job(I'm running spark 1.3.1 with standalone mode on
windows):
./bin/spark-submit \
--class org.apache.spark.examples.SparkPi \
--master spark://localhost:7077 \
--deploy-mode cluster
Hdfs://localhost:443/spark-examples-1.3.1-hadoop2.4.0.jar \
1000
Any ideas about the error?
I've found a similar error in JIRA
https://issues.apache.org/jira/browse/SPARK-1407 but It only occurred at
FileLogger when using yarn and eventlog set to HDFS. In my case, I use
standalone mode and event log set to local, and my error is caused by
Hadoop.util.Shell.runCommand.
Best Regards
Dong Lei
