I did some digging and SparkContext.addJar seems to propagate any file into
the classloader on the executors. That seems to have fixed my problem.
It might also be a way to push log4j.properties or log4j2.xml (depending on
what version you are using) down to the executors.
On Wed, Dec 2, 2015 at
I still have to propagate the file into the directory somehow, and also
that's marked as only for legacy jobs (deprecated?), so no, I have not
experimented with it yet.
On Wed, Dec 2, 2015 at 12:53 AM Rishi Mishra wrote:
> Did you try to use *spark.executor.extraClassPath*. The classpath
> resou
On Tue, Dec 1, 2015 at 12:45 PM, Charles Allen
wrote:
> Is there a way to pass configuration file resources to be resolvable through
> the classloader?
Not in general. If you're using YARN, you can cheat and use
"spark.yarn.dist.files" which will place those files in the classpath;
the same for "
Did you try to use *spark.executor.extraClassPath*. The classpath resources
will be accessible through the executors class loader which executes your
job.
On Wed, Dec 2, 2015 at 2:15 AM, Charles Allen wrote:
> Is there a way to pass configuration file resources to be resolvable
> through the cla
Is there a way to pass configuration file resources to be resolvable
through the classloader?
For example, if I'm using a library (non-spark) that can use a
some-lib.properties file in the classpath/classLoader, can I pass that file
so that when it tries to get the resource from the classloader it
