GitHub user JoshRosen opened a pull request:
https://github.com/apache/spark/pull/2951
[SPARK-3266] [WIP] Remove implementations from Java*Like traits.
This PR addresses a Scala compiler bug
([SI-8905](https://issues.scala-lang.org/browse/SI-8905)) that was breaking
some of the Spark Java APIs. In a nutshell, it seems that methods whose
implementations are inherited from generic traits sometimes have their type
parameters erased to Object. This was causing methods like `DoubleRDD.min()`
to throw confusing NoSuchMethodErrors at runtime.
The workaround is to move the implementations of these methods out of the
trait. Ideally, I'd like to just delete the trait and convert it into an
abstract class, but this technically breaks binary compatibility because it
removes an interface (see #2186). Instead, I remove the method implementations
from the traits and put them in AbstractJavaDStream and AbstractJavaRDD classes
that inherit from their respective traits. This leads to a bit of redundancy,
but I think it's the cleanest general solution that maintains binary
compatibility.
I also improved the test coverage of the Java API.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/JoshRosen/spark javarddlike-trait-bug-fix
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/2951.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2951
----
commit 1b69b474da7732b45d04105e1aa719f0fe6375a4
Author: Josh Rosen <[email protected]>
Date: 2014-10-26T22:42:46Z
[SPARK-3266] Remove implementations from Java*Like traits.
This PR addresses a Scala compiler bug
([SI-8905](https://issues.scala-lang.org/browse/SI-8905)) that was breaking
some of the Spark Java APIs. In a nutshell, it seems that methods whose
implementations are inherited from generic traits sometimes have their type
parameters erased to Object. This was causing methods like `DoubleRDD.min()`
to throw confusing NoSuchMethodErrors at runtime.
The workaround is to move the implementations of these methods out of the
trait. Ideally, I'd like to just delete the trait and convert it into an
abstract class, but this technically breaks binary compatibility because it
removes an interface (see #2186). Instead, I remove the method implementations
from the traits and put them in AbstractJavaDStream and AbstractJavaRDD classes
that inherit from their respective traits. This leads to a bit of redundancy,
but I think it's the cleanest general solution that maintains binary
compatibility.
I also improved the test coverage of the Java API.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
