I realize I'm not looking at all angles here, but the biggest hitch that I see with including both serializers and a migration process is that it requires Flink and its users to stay on JDK < 17 for at least one "bump" version before moving up. At this time they can upgrade to the version of Flink without Kryo v2, built with JDK 17. It's a rather painful request to ask customers to take on.
This feels like the process needs to live outside of Flink so a seamless upgrade path can be supported for both customers and developers. On Thu, Mar 30, 2023 at 8:27 AM Martijn Visser <martijnvis...@apache.org> wrote: > Hi all, > > I also saw a thread on this topic from Clayton Wohl [1] on this topic, > which I'm including in this discussion thread to avoid that it gets lost. > > From my perspective, there's two main ways to get to Java 17: > > 1. The Flink community agrees that we upgrade Kryo to a later version, > which means breaking all checkpoint/savepoint compatibility and releasing a > Flink 2.0 with Java 17 support added and Java 8 and Flink Scala API support > dropped. This is probably the quickest way, but would still mean that we > expose Kryo in the Flink APIs, which is the main reason why we haven't been > able to upgrade Kryo at all. > 2. There's a contributor who makes a contribution that bumps Kryo, but > either a) automagically reads in all old checkpoints/savepoints in using > Kryo v2 and writes them to new snapshots using Kryo v5 (like is mentioned > in the Kryo migration guide [2][3] or b) provides an offline tool that > allows users that are interested in migrating their snapshots manually > before starting from a newer version. That potentially could prevent the > need to introduce a new Flink major version. In both scenarios, ideally the > contributor would also help with avoiding the exposure of Kryo so that we > will be in a better shape in the future. > > It would be good to get the opinion of the community for either of these > two options, or potentially for another one that I haven't mentioned. If it > appears that there's an overall agreement on the direction, I would propose > that a FLIP gets created which describes the entire process. > > Looking forward to the thoughts of others, including the Users (therefore > including the User ML). > > Best regards, > > Martijn > > [1] https://lists.apache.org/thread/qcw8wy9dv8szxx9bh49nz7jnth22p1v2 > [2] https://lists.apache.org/thread/gv49jfkhmbshxdvzzozh017ntkst3sgq > [3] https://github.com/EsotericSoftware/kryo/wiki/Migration-to-v5 > > On Sun, Mar 19, 2023 at 8:16 AM Tamir Sagi <tamir.s...@niceactimize.com> > wrote: > > > I agree, there are several options to mitigate the migration from v2 to > v5. > > yet, Oracle roadmap is to end JDK 11 support in September this year. > > > > > > > > ________________________________ > > From: ConradJam <jam.gz...@gmail.com> > > Sent: Thursday, March 16, 2023 4:36 AM > > To: dev@flink.apache.org <dev@flink.apache.org> > > Subject: Re: [Discussion] - Release major Flink version to support JDK 17 > > (LTS) > > > > EXTERNAL EMAIL > > > > > > > > Thanks for your start this discuss > > > > > > I have been tracking this problem for a long time, until I saw a > > conversation in ISSUSE a few days ago and learned that the Kryo version > > problem will affect the JDK17 compilation of snapshots [1] FLINK-24998 , > > > > As @cherry said it ruined our whole effort towards JDK17 > > > > I am in favor of providing an external tool to migrate from Kryo old > > version checkpoint to the new Kryo new checkpoint at one time (Maybe this > > tool start in flink 2.0 ?), does this tool currently have any plans or > > ideas worth discuss > > > > > > I think it should not be difficult to be compatible with JDK11 and JDK17. > > We should indeed abandon JDK8 in 2.0.0. It is also mentioned in the doc > > that it is marked as Deprecated [2] > > > > > > Here I add that we need to pay attention to the version of Scala and the > > version of JDK17 > > > > > > [1] FLINK-24998 IGSEGV in Kryo / C2 CompilerThread on Java 17 > > https://issues.apache.org/jira/browse/FLINK-24998 > > > > [2] FLINK-30501 Update Flink build instruction to deprecate Java 8 > instead > > of requiring Java 11 https://issues.apache.org/jira/browse/FLINK-30501 > > > > Tamir Sagi <tamir.s...@niceactimize.com> 于2023年3月16日周四 00:54写道： > > > > > Hey dev community, > > > > > > I'm writing this email to kick off a discussion following this epic: > > > FLINK-15736<https://issues.apache.org/jira/browse/FLINK-15736>. > > > > > > We are moving towards JDK 17 (LTS) , the only blocker now is Flink > which > > > currently remains on JDK 11 (LTS). Flink does not support JDK 17 yet, > > with > > > no timeline, the reason, based on the aforementioned ticket is the > > > following tickets > > > > > > 1. FLINK-24998 - SIGSEGV in Kryo / C2 CompilerThread on Java 17< > > > https://issues.apache.org/jira/browse/FLINK-24998>. > > > 2. FLINK-3154 - Update Kryo version from 2.24.0 to latest Kryo LTS > > > version<https://issues.apache.org/jira/browse/FLINK-3154> > > > > > > My question is whether it is possible to release a major version (Flink > > > 2.0.0) using the latest Kryo version for those who don't need to > restore > > > old savepoints/checkpoints in newer format. > > > > > > 1. Leverage JDK 17 features within JVM > > > 2. Moving from the old format to the newer one will be handled only > > > once - a mitigation can be achieved by a conversion tool or external > > > serializers, both can be provided later on. > > > > > > I'd like to emphasize that the next JDK LTS (21) will be released this > > > September. furthermore, Flink already supports JDK 12-15, which is > very > > > close to JDK 17 (LTS) - that was released in September 2021. JDK 11 > will > > > become a legacy soon, as more frameworks moving towards JDK 17 and are > > less > > > likely to support JDK 11 in the near future. (For example, Spring Boot > 3 > > > requires JDK 17 already). > > > > > > Thank you for your consideration of my request. > > > > > > Tamir. > > > > > > > > > > > > > > > Confidentiality: This communication and any attachments are intended > for > > > the above-named persons only and may be confidential and/or legally > > > privileged. Any opinions expressed in this communication are not > > > necessarily those of NICE Actimize. If this communication has come to > you > > > in error you must take no action based on it, nor must you copy or show > > it > > > to anyone; please delete/destroy and inform the sender by e-mail > > > immediately. > > > Monitoring: NICE Actimize may monitor incoming and outgoing e-mails. > > > Viruses: Although we have taken steps toward ensuring that this e-mail > > and > > > attachments are free from any virus, we advise that in keeping with > good > > > computing practice the recipient should ensure they are actually virus > > free. > > > > > > > > > -- > > Best > > > > ConradJam > > > > Confidentiality: This communication and any attachments are intended for > > the above-named persons only and may be confidential and/or legally > > privileged. Any opinions expressed in this communication are not > > necessarily those of NICE Actimize. If this communication has come to you > > in error you must take no action based on it, nor must you copy or show > it > > to anyone; please delete/destroy and inform the sender by e-mail > > immediately. > > Monitoring: NICE Actimize may monitor incoming and outgoing e-mails. > > Viruses: Although we have taken steps toward ensuring that this e-mail > and > > attachments are free from any virus, we advise that in keeping with good > > computing practice the recipient should ensure they are actually virus > free. > > >
