I ran into this problem too and I know of a workaround but don't exactly know what is happening. The work around is to explicitly add either the commons math jar or your application jar (shaded with commons math) to spark.executor.extraClassPath.
My hunch is that this is related to the class loader problem described in [1] where Spark loads breeze at the beginning and then having commons math in the user's jar somehow doesn't get picked up. Thanks Shivaram [1] http://apache-spark-user-list.1001560.n3.nabble.com/Native-library-can-not-be-loaded-when-using-Mllib-PCA-td7042.html#a8307 On Sat, Nov 8, 2014 at 1:21 PM, Sean Owen <[email protected]> wrote: > This means you haven't actually included commons-math3 in your > application. Check the contents of your final app jar and then go > check your build file again. > > On Sat, Nov 8, 2014 at 12:20 PM, lev <[email protected]> wrote: > > Hi, > > I'm using breeze.stats.distributions.Binomial with spark 1.1.0 and having > > the same error. > > I tried to add the dependency to math3 with versions 3.11, 3.2, 3.3 and > it > > didn't help. > > > > Any ideas what might be the problem? > > > > Thanks, > > Lev. > > > > > > anny9699 wrote > >> I use the breeze.stats.distributions.Bernoulli in my code, however met > >> this problem > >> java.lang.NoClassDefFoundError: > >> org/apache/commons/math3/random/RandomGenerator > > > > > > > > > > > > -- > > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/org-apache-commons-math3-random-RandomGenerator-issue-tp15748p18406.html > > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: [email protected] > > For additional commands, e-mail: [email protected] > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > >
