it is mainly a problem because for reasons of sanity one wants to keep single kryo/chill version, and kryo/chill could be used in other places for somewhat persistent serialization by the user.
i know, this is not spark's problem... it is the users problem. but i would find it odd to change kryo in a minor upgrade in general. not that it cannot be done. On Fri, Jan 19, 2018 at 8:55 AM, Sean Owen <so...@cloudera.com> wrote: > See: > > https://issues.apache.org/jira/browse/SPARK-23131 > https://github.com/apache/spark/pull/20301#issuecomment-358473199 > > I expected a major Kryo upgrade to be problematic, but it worked fine. It > picks up a number of fixes: > https://github.com/EsotericSoftware/kryo/releases/tag/kryo-parent-4.0.0 > > It might be good for Spark 2.4. > > Its serialized format isn't entirely compatible though. I'm trying to > recall whether this is a problem in practice. We don't guarantee wire > compatibility across mismatched Spark versions, right? > > But does the Kryo serialized form show up in any persistent stored form? I > don't believe any normal output, even that of saveAsObjectFile, uses it. > > I'm wondering if I am not recalling why this would be a problem to update? > > Sean >
