Sure :-) Digging into the TargetInvocationException revealed a “NoSuchFieldError: DEFAULT_MAX_PENDING_TASKS", which we guessed was linked to some kind of binary incompatibility in the dependencies. Looking into the stack trace this could be traced to a dynamic constructor call in netty, and we could see that Sparks netty-all dependency had been lifted from 4.0.29 to 4.0.42. At the same time we have a dependency on netty-transport 4.0.37 due to our use of the Cassandra Java Driver, and those classes are also in netty-all. Looking into the versions we identified that these were indeed not binary compatible - a field had been removed as the original error said. Since the artifacts are different they can co-exist on the classpath, but of course not at runtime, and we then get the error when incompatible versions are loaded. Interestingly we could not reproduce this on our laptops - only on our buildserver - so there must be some random difference in classloader behaviour.
Dependency issues is the typical problem we have when taking in a new version of Spark and they always take some time to find out. We had a lot of issues with Guava when taking in 2.0 and ended up shading it when no other option was available. It would be really nice if the version bumps were included in the release notes. Morten > Den 20. jan. 2017 kl. 20.21 skrev Jacek Laskowski <ja...@japila.pl>: > > Hi, > > I'd be very interested in how you figured it out. Mind sharing? > > Jacek > > On 18 Jan 2017 9:51 p.m., "mhornbech" <mor...@datasolvr.com > <mailto:mor...@datasolvr.com>> wrote: > For anyone revisiting this at a later point, the issue was that Spark 2.1.0 > upgrades netty to version 4.0.42 which is not binary compatible with version > 4.0.37 used by version 3.1.0 of the Cassandra Java Driver. The newer version > can work with Cassandra, but because of differences in the maven artifacts > (Spark depends on netty-all while Cassandra depends on netty-transport) this > was not automatically resolved by SBT. Adding an explicit dependency to > netty-transport version 4.0.42 solved the problem. > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/New-runtime-exception-after-switch-to-Spark-2-1-0-tp28263p28319.html > > <http://apache-spark-user-list.1001560.n3.nabble.com/New-runtime-exception-after-switch-to-Spark-2-1-0-tp28263p28319.html> > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > --------------------------------------------------------------------- > To unsubscribe e-mail: user-unsubscr...@spark.apache.org > <mailto:user-unsubscr...@spark.apache.org> >
