We should consider this for Spark 2.0.
On Wed, Nov 11, 2015 at 2:01 PM, Steve Loughran <ste...@hortonworks.com> wrote: > > > Spark is currently on a fairly dated version of Kryo 2.x; it's trailing on > the fixes in Hive and, as the APIs are incompatible, resulted in that > mutant spark-project/hive JAR needed for the Hive 1.2.1 support > > But: updating it hasn't been an option, because Spark needs to be in sync > with Twitter's Chill library. > > There's now an offer from Twitter to help coordinate a kryo update across > Chill, Scalding and other things they use > > https://github.com/twitter/chill/pull/230 > > Given kryo is "The guava jar of serialization", I doubt anyone is jumping > up and down wanting this, but it is something to consider. Once hive moves > to it, all the hive spark integration is probably going to break again; > getting in sync with hive (see SPARK-10793) would reduce the traumaticness > of hive updates >
