Problem is noted here: https://issues.apache.org/jira/browse/SPARK-5949
I tried this as a workaround:
import org.apache.spark.scheduler._
import org.roaringbitmap._
...
kryo.register(classOf[org.roaringbitmap.RoaringBitmap])
kryo.register(classOf[org.roaringbitmap.RoaringArray])
kryo.register(classOf[org.roaringbitmap.ArrayContainer])
kryo.register(classOf[org.apache.spark.scheduler.HighlyCompressedMapStatus])
kryo.register(classOf[org.roaringbitmap.RoaringArray$Element])
kryo.register(classOf[org.roaringbitmap.RoaringArray$Element[]])
kryo.register(classOf[short[]])
in build file:
libraryDependencies += "org.roaringbitmap" % "RoaringBitmap" % "0.4.8"
This fails to compile:
...:53: identifier expected but ']' found.
[error] kryo.register(classOf[org.roaringbitmap.RoaringArray$Element[]])
also:
:54: identifier expected but ']' found.
[error] kryo.register(classOf[short[]])
also:
:51: class HighlyCompressedMapStatus in package scheduler cannot be
accessed in package org.apache.spark.scheduler
[error]
kryo.register(classOf[org.apache.spark.scheduler.HighlyCompressedMapStatus])
Suggestions?
Arun