Hi,
I am running spark-1.1.0 and I want to pass in a file (that contains java
serialized objects used to initialize my program) to the App main program. I
am using the --files option but I am not able to retrieve the file in the
main_class. It reports a null pointer exception. [I tried both local &
yarn-cluster with the same result]. I am using the
SparkFiles.get("myobject.ser") to get the file. Am I doing something wrong ?
CMD:
bin/spark-submit --name Test --class
com.test.batch.modeltrainer.ModelTrainerMain \
--master local --files /tmp/myobject.ser --verbose
/opt/test/lib/spark-test.jar
com.test.batch.modeltrainer.ModelTrainerMain.scala
37: val serFile = SparkFiles.get("myobject.ser")
Exception:
Exception in thread "main" java.lang.NullPointerException
at org.apache.spark.SparkFiles$.getRootDirectory(SparkFiles.scala:37)
at org.apache.spark.SparkFiles$.get(SparkFiles.scala:31)
at
com.test.batch.modeltrainer.ModelTrainerMain$.main(ModelTrainerMain.scala:37)
at
com.test.batch.modeltrainer.ModelTrainerMain.main(ModelTrainerMain.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:303)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:55)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Looking at the Scala code for SparkFiles:37, it looks like SparkEnv.get is
getting a null ..
Thanks
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/spark-submit-command-line-with-files-tp14645.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
