I might be stating the obvious for everyone, but the issue here is not
reflection or the source of the JAR, but the ClassLoader. The basic
rules are this.
"new Foo" will use the ClassLoader that defines Foo. This is usually
the ClassLoader that loaded whatever it is that first referenced Foo
and caused it to be loaded -- usually the ClassLoader holding your
other app classes.
ClassLoaders can have a parent-child relationship. ClassLoaders always
look in their parent before themselves.
(Careful then -- in contexts like Hadoop or Tomcat where your app is
loaded in a child ClassLoader, and you reference a class that Hadoop
or Tomcat also has (like a lib class) you will get the container's
version!)
When you load an external JAR it has a separate ClassLoader which does
not necessarily bear any relation to the one containing your app
classes, so yeah it is not generally going to make "new Foo" work.
Reflection lets you pick the ClassLoader, yes.
I would not call setContextClassLoader.
On Mon, May 19, 2014 at 12:00 AM, Sandy Ryza <sandy.r...@cloudera.com> wrote:
> I spoke with DB offline about this a little while ago and he confirmed that
> he was able to access the jar from the driver.
>
> The issue appears to be a general Java issue: you can't directly
> instantiate a class from a dynamically loaded jar.
>
> I reproduced it locally outside of Spark with:
> ---
> URLClassLoader urlClassLoader = new URLClassLoader(new URL[] { new
> File("myotherjar.jar").toURI().toURL() }, null);
> Thread.currentThread().setContextClassLoader(urlClassLoader);
> MyClassFromMyOtherJar obj = new MyClassFromMyOtherJar();
> ---
>
> I was able to load the class with reflection.
