Yes here is why the initial effort didnt work, explained a bit better. As I mentioned earlier SparkContext will add your jars/files (declared with the related conf properties) to the FileServer. If it is a local to the container's fs jar (has schema local:) it will just be resolved to: file + absolute path ( https://github.com/apache/spark/blob/20a3ef7259490e0c9f6348f13db1e99da5f0df83/core/src/main/scala/org/apache/spark/SparkContext.scala#L1805-L1814). Check:
// A JAR file which exists only on the driver node case null => // SPARK-22585 path without schema is not url encoded addJarFile(new File(uri.getRawPath)) // A JAR file which exists only on the driver node case "file" => addJarFile(new File(uri.getPath)) // A JAR file which exists locally on every worker node case "local" => "file:" + uri.getPath case _ => path That means the task that will run at the executor side, when task description will be de-serialized, will have a uri to resolve that starts with file://. That means updateDependencies at the executor side will call doFetchFile and will choose this path: case "file" => // In the case of a local file, copy the local file to the target directory. // Note the difference between uri vs url. val sourceFile = if (uri.isAbsolute) new File(uri) else new File(url) copyFile(url, sourceFile, targetFile, fileOverwrite) But since the file does not exist on your executor image it will fail. The solution you have works because you initially pass a file://path uri that the SparkContext will add from the driver's fs to the SparkContext fileserver, meaning it will be available at the executors side from a new uri with schema spark:// (there is a call to addJarFile while for local:// there is not). Hope this helps. Stavros On Sun, Apr 28, 2019 at 7:29 AM Nikhil Chinnapa <nishant.ran...@renovite.com> wrote: > Hi Stavros, > > Thanks a lot for pointing in right direction. I got stuck in some release, > so didn’t got time earlier. > > The mistake was “LINUX_APP_RESOURCE” : I was using “local” instead it > should > be “file”. I reached above due to your email only. > > What I understood: > Driver image : $SPARK_HOME/bin and $SPARK_HOME/jars and application jar. > Executor Image : just $SPARK_HOME/bin and $SPARK_HOME/jars folder will > suffice. > > > > > -- > Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/ > > --------------------------------------------------------------------- > To unsubscribe e-mail: user-unsubscr...@spark.apache.org > >
