Thanks Chesnay for working on this. Would you like to share more info about the JDK bug?
Best regards, Jing On Mon, Apr 24, 2023 at 11:39 AM Chesnay Schepler <ches...@apache.org> wrote: > As it turns out Kryo isn't a blocker; we ran into a JDK bug. > > On 31/03/2023 08:57, Chesnay Schepler wrote: > > > https://github.com/EsotericSoftware/kryo/wiki/Migration-to-v5#migration-guide > > Kroy themselves state that v5 likely can't read v2 data. > > However, both versions can be on the classpath without classpath as v5 > offers a versioned artifact that includes the version in the package. > > It probably wouldn't be difficult to migrate a savepoint to Kryo v5, > purely from a read/write perspective. > > The bigger question is how we expose this new Kryo version in the API. If > we stick to the versioned jar we need to either duplicate all current > Kryo-related APIs or find a better way to integrate other serialization > stacks. > On 30/03/2023 17:50, Piotr Nowojski wrote: > > Hey, > > > 1. The Flink community agrees that we upgrade Kryo to a later version, > which means breaking all checkpoint/savepoint compatibility and releasing a > Flink 2.0 with Java 17 support added and Java 8 and Flink Scala API support > dropped. This is probably the quickest way, but would still mean that we > expose Kryo in the Flink APIs, which is the main reason why we haven't been > able to upgrade Kryo at all. > > This sounds pretty bad to me. > > Has anyone looked into what it would take to provide a smooth migration > from Kryo2 -> Kryo5? > > Best, > Piotrek > > czw., 30 mar 2023 o 16:54 Alexis Sarda-Espinosa <sarda.espin...@gmail.com> > napisał(a): > >> Hi Martijn, >> >> just to be sure, if all state-related classes use a POJO serializer, Kryo >> will never come into play, right? Given FLINK-16686 [1], I wonder how many >> users actually have jobs with Kryo and RocksDB, but even if there aren't >> many, that still leaves those who don't use RocksDB for >> checkpoints/savepoints. >> >> If Kryo were to stay in the Flink APIs in v1.X, is it impossible to let >> users choose between v2/v5 jars by separating them like log4j2 jars? >> >> [1] https://issues.apache.org/jira/browse/FLINK-16686 >> >> Regards, >> Alexis. >> >> Am Do., 30. März 2023 um 14:26 Uhr schrieb Martijn Visser < >> martijnvis...@apache.org>: >> >>> Hi all, >>> >>> I also saw a thread on this topic from Clayton Wohl [1] on this topic, >>> which I'm including in this discussion thread to avoid that it gets lost. >>> >>> From my perspective, there's two main ways to get to Java 17: >>> >>> 1. The Flink community agrees that we upgrade Kryo to a later version, >>> which means breaking all checkpoint/savepoint compatibility and releasing a >>> Flink 2.0 with Java 17 support added and Java 8 and Flink Scala API support >>> dropped. This is probably the quickest way, but would still mean that we >>> expose Kryo in the Flink APIs, which is the main reason why we haven't been >>> able to upgrade Kryo at all. >>> 2. There's a contributor who makes a contribution that bumps Kryo, but >>> either a) automagically reads in all old checkpoints/savepoints in using >>> Kryo v2 and writes them to new snapshots using Kryo v5 (like is mentioned >>> in the Kryo migration guide [2][3] or b) provides an offline tool that >>> allows users that are interested in migrating their snapshots manually >>> before starting from a newer version. That potentially could prevent the >>> need to introduce a new Flink major version. In both scenarios, ideally the >>> contributor would also help with avoiding the exposure of Kryo so that we >>> will be in a better shape in the future. >>> >>> It would be good to get the opinion of the community for either of these >>> two options, or potentially for another one that I haven't mentioned. If it >>> appears that there's an overall agreement on the direction, I would propose >>> that a FLIP gets created which describes the entire process. >>> >>> Looking forward to the thoughts of others, including the Users >>> (therefore including the User ML). >>> >>> Best regards, >>> >>> Martijn >>> >>> [1] https://lists.apache.org/thread/qcw8wy9dv8szxx9bh49nz7jnth22p1v2 >>> [2] https://lists.apache.org/thread/gv49jfkhmbshxdvzzozh017ntkst3sgq >>> [3] https://github.com/EsotericSoftware/kryo/wiki/Migration-to-v5 >>> >>> On Sun, Mar 19, 2023 at 8:16 AM Tamir Sagi <tamir.s...@niceactimize.com> >>> wrote: >>> >>>> I agree, there are several options to mitigate the migration from v2 to >>>> v5. >>>> yet, Oracle roadmap is to end JDK 11 support in September this year. >>>> >>>> >>>> >>>> ________________________________ >>>> From: ConradJam <jam.gz...@gmail.com> >>>> Sent: Thursday, March 16, 2023 4:36 AM >>>> To: dev@flink.apache.org <dev@flink.apache.org> >>>> Subject: Re: [Discussion] - Release major Flink version to support JDK >>>> 17 (LTS) >>>> >>>> EXTERNAL EMAIL >>>> >>>> >>>> >>>> Thanks for your start this discuss >>>> >>>> >>>> I have been tracking this problem for a long time, until I saw a >>>> conversation in ISSUSE a few days ago and learned that the Kryo version >>>> problem will affect the JDK17 compilation of snapshots [1] FLINK-24998 , >>>> >>>> As @cherry said it ruined our whole effort towards JDK17 >>>> >>>> I am in favor of providing an external tool to migrate from Kryo old >>>> version checkpoint to the new Kryo new checkpoint at one time (Maybe >>>> this >>>> tool start in flink 2.0 ?), does this tool currently have any plans or >>>> ideas worth discuss >>>> >>>> >>>> I think it should not be difficult to be compatible with JDK11 and >>>> JDK17. >>>> We should indeed abandon JDK8 in 2.0.0. It is also mentioned in the doc >>>> that it is marked as Deprecated [2] >>>> >>>> >>>> Here I add that we need to pay attention to the version of Scala and the >>>> version of JDK17 >>>> >>>> >>>> [1] FLINK-24998 IGSEGV in Kryo / C2 CompilerThread on Java 17 >>>> https://issues.apache.org/jira/browse/FLINK-24998 >>>> >>>> [2] FLINK-30501 Update Flink build instruction to deprecate Java 8 >>>> instead >>>> of requiring Java 11 https://issues.apache.org/jira/browse/FLINK-30501 >>>> >>>> Tamir Sagi <tamir.s...@niceactimize.com> 于2023年3月16日周四 00:54写道： >>>> >>>> > Hey dev community, >>>> > >>>> > I'm writing this email to kick off a discussion following this epic: >>>> > FLINK-15736<https://issues.apache.org/jira/browse/FLINK-15736>. >>>> > >>>> > We are moving towards JDK 17 (LTS) , the only blocker now is Flink >>>> which >>>> > currently remains on JDK 11 (LTS). Flink does not support JDK 17 yet, >>>> with >>>> > no timeline, the reason, based on the aforementioned ticket is the >>>> > following tickets >>>> > >>>> > 1. FLINK-24998 - SIGSEGV in Kryo / C2 CompilerThread on Java 17< >>>> > https://issues.apache.org/jira/browse/FLINK-24998>. >>>> > 2. FLINK-3154 - Update Kryo version from 2.24.0 to latest Kryo LTS >>>> > version<https://issues.apache.org/jira/browse/FLINK-3154> >>>> > >>>> > My question is whether it is possible to release a major version >>>> (Flink >>>> > 2.0.0) using the latest Kryo version for those who don't need to >>>> restore >>>> > old savepoints/checkpoints in newer format. >>>> > >>>> > 1. Leverage JDK 17 features within JVM >>>> > 2. Moving from the old format to the newer one will be handled only >>>> > once - a mitigation can be achieved by a conversion tool or external >>>> > serializers, both can be provided later on. >>>> > >>>> > I'd like to emphasize that the next JDK LTS (21) will be released this >>>> > September. furthermore, Flink already supports JDK 12-15, which is >>>> very >>>> > close to JDK 17 (LTS) - that was released in September 2021. JDK 11 >>>> will >>>> > become a legacy soon, as more frameworks moving towards JDK 17 and >>>> are less >>>> > likely to support JDK 11 in the near future. (For example, Spring >>>> Boot 3 >>>> > requires JDK 17 already). >>>> > >>>> > Thank you for your consideration of my request. >>>> > >>>> > Tamir. >>>> > >>>> > >>>> > >>>> > >>>> > Confidentiality: This communication and any attachments are intended >>>> for >>>> > the above-named persons only and may be confidential and/or legally >>>> > privileged. Any opinions expressed in this communication are not >>>> > necessarily those of NICE Actimize. If this communication has come to >>>> you >>>> > in error you must take no action based on it, nor must you copy or >>>> show it >>>> > to anyone; please delete/destroy and inform the sender by e-mail >>>> > immediately. >>>> > Monitoring: NICE Actimize may monitor incoming and outgoing e-mails. >>>> > Viruses: Although we have taken steps toward ensuring that this >>>> e-mail and >>>> > attachments are free from any virus, we advise that in keeping with >>>> good >>>> > computing practice the recipient should ensure they are actually >>>> virus free. >>>> > >>>> >>>> >>>> -- >>>> Best >>>> >>>> ConradJam >>>> >>>> Confidentiality: This communication and any attachments are intended >>>> for the above-named persons only and may be confidential and/or legally >>>> privileged. Any opinions expressed in this communication are not >>>> necessarily those of NICE Actimize. If this communication has come to you >>>> in error you must take no action based on it, nor must you copy or show it >>>> to anyone; please delete/destroy and inform the sender by e-mail >>>> immediately. >>>> Monitoring: NICE Actimize may monitor incoming and outgoing e-mails. >>>> Viruses: Although we have taken steps toward ensuring that this e-mail >>>> and attachments are free from any virus, we advise that in keeping with >>>> good computing practice the recipient should ensure they are actually virus >>>> free. >>>> >>> > >
