Github user DieBauer commented on the issue:
https://github.com/apache/flink/pull/3703
Ok, started looking into the issue a bit more.
It seems like it has to do with the new lambda generation in scala 2.12 and
not with the asm library.
From the scala 2.12-M3 release notes
(https://github.com/scala/scala/releases/tag/v2.12.0-M3):
> Java 8 style closure classes
>
> Scala 2.12 emits closures in the same style as Java 8.
>
> For each lambda the compiler generates a method containing the lambda
body.
> At runtime, this method is passed as an argument to the LambdaMetaFactory
provided by the JDK, which creates a closure object.
>
> Compared to Scala 2.11, the new scheme has the advantage that the
compiler does not generate an anonymous class for each lambda anymore.
> This leads to significantly smaller JAR files.
Our ClosureCleaner uses the class name for instantiating the ClassReader,
which is used later on.
However, since scala2.12 doesn't generate anonymous classes, the file isn't
found (null), therefore we get class not found exception, which make sense now.
We have to look into how to circumvent/implement this new generation of
'lambdas'.
A small technical example, the testclass which throwed an exception
`AcceptPFTestBase`.
And then the line containing: `protected val groupedTuples =
tuples.groupBy(_._1)`
Since tuples is a `Dataset` the function that we have to check is `_._1`
(an anonymous function).
when compiling/executing with scala 2.11
we get `class
org.apache.flink.api.scala.extensions.base.AcceptPFTestBase$$anonfun$1` as
`cls` in the `ClosureCleaner.getClassReader` method.
And this is indeed a file generated by the scala compiler and can be
resolved by `ls.getResourceAsStream(className)`.
However when using scala 2.12
we get `class
org.apache.flink.api.scala.extensions.base.AcceptPFTestBase$$Lambda$11/1489743810`
which is not an existing file, and cannot be resolved by
`ls.getResourceAsStream(className)`.
Concluding, according to me, with the new scala 2.12 style with lambdas,
the current closurecleaner doesn't suffice.
There is also a Spark issue
(https://issues.apache.org/jira/browse/SPARK-14540) regarding closures in scala
2.12/java8.
Any thoughts how to proceed?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
