hi all,
we are switching from scala 2.11 to 2.12 with a spark 2.4.1 release
candidate and so far this has been going pretty smoothly.
however we do see some new serialization errors related to Function1,
Function2, etc.
they look like this:
ClassCastException: cannot assign instance of
java.lang.invoke.SerializedLambda to field MyCaseClass.f of type
scala.Function1 in instance of MyCaseClass
at
java.io.ObjectStreamClass$FieldReflector.setObjFieldValues(ObjectStreamClass.java:2133)
these are often simple case classes with inside a val for the function.
like this:
case class MyCaseClass[X, Y](...) {
val f: Function1[X, Y] = ...
}
we had no problems with these in scala 2.11. it does not look like these
classes have members that are not serializable, and neither do the
functions close over anything troublesome. since we get this for some
classes but not for others i am not entirely sure what to make of it. we
can work around the issue by changing the val f to a def, like this:
case class MyCaseClass[X, Y](...) {
def f: Function1[X, Y] = ...
}
any idea what is causing this?
thanks!
koert
