Perhaps we should provide the Backward compatible version in the initial 2.0 release, and provided a deprecation plan for simplifying the code and only supporting the newer Kryo 5.x in Flink 2.1+. This would be inline with the previous decision provided in the FLIP's compatibility and deprecation section: https://cwiki.apache.org/confluence/display/FLINK/FLIP-317:+Upgrade+Kryo+from+2.24.0+to+5.5.0
>From the FLIP: "All Flink applications currently generate state with Kryo v2. This PR will fully support reading that state and write newer state with Kryo v5. However, some future version of Flink would eventually drop support for Kryo v2 and users will need to make sure all persisted state is upgraded before upgrading to a future version of Flink that drops support for Kryo v2 serialized state. There will be a migration pathway so that all applications can upgrade without losing state. Applications will have to run on a bridge release version that will read their state with Kryo v2 data and write it with Kryo v5 data before upgrading to a future version of Flink that completely drops support for Kryo v2." Another distinction between PR #1 and PR #2 relates to the Twitter Chill dependency. I would recommend we still remove the twitter/chill dependency. This project is no longer maintained and we were not able to get our Kryo 5.x update merged in. Nick On 2025/02/24 11:58:35 Gyula Fóra wrote: > Thanks for chiming in Timo, I completely agree. > > There seem to be 2 approaches available here: > 1. Backward compatible: https://github.com/apache/flink/pull/22660 > 2. Non compatible https://github.com/apache/flink/pull/25896 > > They both seem to be feasible however the backward compatible approach is > much more complex overall. I don't really have experience with the Kryo > logic to make an informed call here. > Would be good to hear from someone with deeper experience with our Kryo > serialization stack. > > Gyula > > On Mon, Feb 24, 2025 at 12:37 PM Timo Walther <twal...@apache.org> wrote: > > > Hi Gyula, > > > > thanks for bringing this up. I agree that we should use one of the most > > recent Kryo versions for Flink 2.0. Otherwise the community will suffer > > again from incompatibilities and needs to wait for another major Flink > > version. Thanks for starting this discussion. We should do it for 2.0. > > If not now, when? > > > > Cheers, > > Timo > > > > On 24.02.25 08:03, Gyula Fóra wrote: > > > Hey! > > > > > > I think the main question here is whether Flink 2.0 is going to be state > > > backward compatible or not. > > > If not then we need to make the upgrade right now, freeze or not. We have > > > to decide this as a community. > > > > > > If we need to preserve backward compatibility then we need to go with a > > > much more complex solution here when upgrading Kryo to do that. > > > > > > Do we have a community decision somewhere related to state > > > backward compatibility for 2.0? > > > > > > Cheers > > > Gyula > > > > > > On Mon, Feb 24, 2025 at 3:02 AM Rui Fan <1996fan...@gmail.com> wrote: > > > > > >> Thanks Gyula for driving this discussion! > > >> > > >> +1 for upgrading kyro, I have a question about the timeline. > > >> The flink 2.0.0 has been freezed, do we have enough time to > > >> test if it's done in flink 2.0.0? > > >> > > >> Best, > > >> Rui > > >> > > >> On Fri, Feb 21, 2025 at 9:55 PM Alexander Fedulov <afedu...@apache.org> > > >> wrote: > > >> > > >>> Hi Gyula, > > >>> > > >>> Thanks for bringing up this topic! Kryo incompatibility with newer > > >>> Java versions is a major issue that needs to be addressed, and in my > > >>> opinion, Flink 2.0 provides a great opportunity to introduce this > > >>> change. > > >>> > > >>> My understanding is that state compatibility was not a strict goal > > >>> during 2.0 development. However, I’ve heard mentions that it might > > >>> actually be compatible after all. > > >>> > > >>> From this perspective: > > >>> definitely +1 on upgrading Kryo. The decision to invest additional > > >>> effort in maintaining compatibility should depend on whether all other > > >>> changes have preserved compatibility guarantees. If Kryo is the only > > >>> breaking change, then ensuring compatibility might be worth > > >>> considering. > > >>> > > >>> Best, > > >>> Alex > > >>> > > >>> On Fri, 21 Feb 2025 at 06:05, Gyula Fóra <gyula.f...@gmail.com> wrote: > > >>>> > > >>>> Hey all! > > >>>> > > >>>> I would like to rekindle this discussion as it seems that it has > > >> stalled > > >>>> several times in the past and we are nearing the point in time where > > >> the > > >>>> decision has to be made with regards to 2.0. (we are already a bit > > late > > >>> but > > >>>> nevermind) > > >>>> > > >>>> There has been numerous requests and efforts to upgrade Kryo to better > > >>>> support newer Java versions and Java native types. I think we can all > > >>> agree > > >>>> that this change is inevitable one way or another. > > >>>> > > >>>> The latest JIRA for this seems to be: > > >>>> https://issues.apache.org/jira/browse/FLINK-3154 > > >>>> > > >>>> There is even an open PR that accomplishes this (currently in a state > > >>>> incompatible way) but based on the discussion it seems that with some > > >>> extra > > >>>> complexity compatibility can even be preserved by having both the old > > >> and > > >>>> new Kryo versions active at the same time. > > >>>> > > >>>> The main question here is whether state compatibility is important for > > >>> 2.0 > > >>>> with this regard or we want to bite the bullet and make this upgrade > > >> once > > >>>> and for all. > > >>>> > > >>>> Cheers, > > >>>> Gyula > > >>> > > >> > > > > > > > >
