i think its probably fine, but i remember updating kryo and chill to be a
major issue with scalding historically exactly because kryo was also used
for serialized data on disk by some major users.

On Fri, Jan 19, 2018 at 12:13 PM, Sean Owen <so...@cloudera.com> wrote:

> Yeah, if users are using Kryo directly, they should be insulated from a
> Spark-side change because of shading.
> However this also entails updating (unshaded) Chill from 0.8.x to 0.9.x. I
> am not sure if that causes problems for apps.
>
> Normally I'd avoid any major-version change in a minor release. This one
> looked potentially entirely internal.
> I think if there are any doubts, we can leave it for Spark 3. There was a
> bug report that needed a fix from Kryo 4, but it might be minor after all.
>
>
> On Fri, Jan 19, 2018 at 11:05 AM Koert Kuipers <ko...@tresata.com> wrote:
>
>> it is mainly a problem because for reasons of sanity one wants to keep
>> single kryo/chill version, and kryo/chill could be used in other places for
>> somewhat persistent serialization by the user.
>>
>> i know, this is not spark's problem... it is the users problem. but i
>> would find it odd to change kryo in a minor upgrade in general. not that it
>> cannot be done.
>>
>>
>>
>> On Fri, Jan 19, 2018 at 8:55 AM, Sean Owen <so...@cloudera.com> wrote:
>>
>>> See:
>>>
>>> https://issues.apache.org/jira/browse/SPARK-23131
>>> https://github.com/apache/spark/pull/20301#issuecomment-358473199
>>>
>>> I expected a major Kryo upgrade to be problematic, but it worked fine.
>>> It picks up a number of fixes:
>>> https://github.com/EsotericSoftware/kryo/releases/tag/kryo-parent-4.0.0
>>>
>>> It might be good for Spark 2.4.
>>>
>>> Its serialized format isn't entirely compatible though. I'm trying to
>>> recall whether this is a problem in practice. We don't guarantee wire
>>> compatibility across mismatched Spark versions, right?
>>>
>>> But does the Kryo serialized form show up in any persistent stored form?
>>> I don't believe any normal output, even that of saveAsObjectFile, uses it.
>>>
>>> I'm wondering if I am not recalling why this would be a problem to
>>> update?
>>>
>>> Sean
>>>
>>
>>

Reply via email to