I agree that 1.1 compatibility is the most important “pain point", as compatibility with the rest of the versions follows a more “systematic” approach.
I think that discarding compatibility with 1.1 will clear some parts of the codebase significantly. Kostas > On Jun 27, 2017, at 6:03 PM, Stephan Ewen <se...@apache.org> wrote: > > I think that this discussion is probably motivated especially by the > "legacy state" handling of Flink 1.1. > The biggest gain in codebase and productivity would be won only by dropping > 1.1 compatibility in Flink 1.4. > > My gut feeling is that this is reasonable. We support two versions back, > which means that users can skip one upgrade, but not two. > > From what I can tell, users are usually eager to upgrade. They don't do it > immediately, but as soon as the new release is a bit battle tested. > > I would expect skipping two entire versions to be rare enough to be okay > with a solution which is a bit more effort for the user: > You can upgrade from Flink 1.1. to 1.4 by loading the 1.1 savepoint into > Flink 1.2, take a savepoint (1.2 format), and resume that in Flink 1.4. > > Greetings, > Stephan > > > On Tue, Jun 27, 2017 at 12:01 PM, Stefan Richter < > s.rich...@data-artisans.com> wrote: > >> For many parts of the code, I would agree with Aljoscha. However, I can >> also see notable exceptions, such as maintaining support for the legacy >> state from Flink <=1.1. For example, I think dropping support for this can >> simplify new developments such as fast local recovery or state replication >> quiet a bit because this is a special case that runs through a lot of code >> from backend to JM. So besides this general discussion about a backwards >> compatible policy, do you think it could make sense to start another >> concrete discussion about if we still must or want backwards compatibility >> to Flink 1.1 in Flink 1.4? >> >>> Am 29.05.2017 um 12:08 schrieb Aljoscha Krettek <aljos...@apache.org>: >>> >>> Normally, I’m the first one to suggest removing everything that is not >> absolutely necessary in order to have a clean code base. On this issue, >> though, I think we should support restoring from old Savepoints as far back >> as possible if it does not make the code completely unmaintainable. Some >> users might jump versions and always forcing them to go though every >> version from their old version to the current version doesn’t seem feasible >> and might put off some users. >>> >>> So far, I think the burden of supporting restore from 1.1 is still small >> enough and with each new version the changes between versions become less >> and less. The changes from 1.2 to the upcoming 1.3 are quite minimal, I >> think. >>> >>> Best, >>> Aljoscha >>>> On 24. May 2017, at 17:58, Ted Yu <yuzhih...@gmail.com> wrote: >>>> >>>> bq. about having LTS versions once a year >>>> >>>> +1 to the above. >>>> >>>> There may be various reasons users don't want to upgrade (after new >>>> releases come out). We should give such users enough flexibility on the >>>> upgrade path. >>>> >>>> Cheers >>>> >>>> On Wed, May 24, 2017 at 8:39 AM, Kostas Kloudas < >> k.klou...@data-artisans.com >>>>> wrote: >>>> >>>>> Hi all, >>>>> >>>>> For the proposal of having a third party tool, I agree with Ted. >>>>> Maintaining >>>>> it is a big and far from trivial effort. >>>>> >>>>> Now for the window of backwards compatibility, I would argue that even >> if >>>>> for some users 4 months (1 release) is not enough to bump their Flink >>>>> version, >>>>> the proposed policy guarantees that there will always be a path from >> any >>>>> old >>>>> version to any subsequent one. >>>>> >>>>> Finally, for the proposal about having LTS versions once a year, I am >> not >>>>> sure if this will reduce or create more overhead. If I understand the >> plan >>>>> correctly, this would mean that the community will have to maintain >>>>> 2 or 3 LTS versions and the last two major ones, right? >>>>> >>>>>> On May 22, 2017, at 7:31 PM, Ted Yu <yuzhih...@gmail.com> wrote: >>>>>> >>>>>> For #2, it is difficult to achieve: >>>>>> >>>>>> a. maintaining savepoint migration is non-trivial and should be >> reviewed >>>>> by >>>>>> domain experts >>>>>> b. how to certify such third-party tool >>>>>> >>>>>> Cheers >>>>>> >>>>>> On Mon, May 22, 2017 at 3:04 AM, 施晓罡 <shixiaoga...@gmail.com> wrote: >>>>>> >>>>>>> Hi all, >>>>>>> >>>>>>> Currently, we work a lot in the maintenance of compatibility. >>>>>>> There exist much code in runtime to support the migration of >> savepoints >>>>>>> (most of which are deprecated), making it hard to focus on the >> current >>>>>>> implementation. >>>>>>> When more versions are released, much more efforts will be needed if >> we >>>>>>> try to make these released versions compatible. >>>>>>> >>>>>>> I agree with Tzu-Li that we should provide a method to let users >> upgrade >>>>>>> Flink in a reasonable pace. >>>>>>> But i am against the proposal that we only offer backwards >> compatibility >>>>>>> for one previous version. >>>>>>> According our time-based release model, a major version is released >>>>> every >>>>>>> four month. >>>>>>> That means, users have to upgrade their versions every 8 months. >>>>> Otherwise >>>>>>> they will have difficulties in the migration of existing savepoints. >>>>>>> >>>>>>> My suggestions include >>>>>>> >>>>>>> (1) We can release Long-Term Support (LTS) versions which are widely >>>>>>> adopted in other open-source projects. >>>>>>> LTS versions should be stable and are free of found bugs. Savepoints >> in >>>>>>> LTS versions are guaranteed to be back-compatible so that users can >>>>> easily >>>>>>> upgrade to newer LTS versions. >>>>>>> >>>>>>> The releasing of LTS versions is slower than that of major versions >>>>> (maybe >>>>>>> once a year, determined by users’ upgrade frequency). >>>>>>> Each LTS version will be supported a period of time and typically >> there >>>>>>> are no more than three active LTS versions. >>>>>>> By encouraging users to use LTS versions, we can ease the >> maintenance of >>>>>>> released versions (bug fixes, back compatibility, and critical >>>>> performance >>>>>>> improvement). >>>>>>> >>>>>>> (2) We can provide a third-party tool to do the migration of >>>>> old-versioned >>>>>>> savepoints. >>>>>>> When users upgrade their versions, they can use the provided tool to >>>>>>> migrate existing savepoints. >>>>>>> This can help move the code for savepoint migration out of the actual >>>>>>> codebase, making code focuses on current implementation. >>>>>>> >>>>>>> What do you think? >>>>>>> >>>>>>> Regards, >>>>>>> Xiaogang >>>>>>> >>>>>>> >>>>>>>> 在 2017年5月22日，下午1:39，Tzu-Li (Gordon) Tai <tzuli...@apache.org> 写道： >>>>>>>> >>>>>>>> Hi Kostas, >>>>>>>> >>>>>>>> Thanks for bringing this up! >>>>>>>> I think it is reasonable to keep this coherent with our timely-based >>>>>>> release model guarantees. >>>>>>>> >>>>>>>> With the timely-based release model, there is a guarantee that the >>>>>>> current latest major version and the previous one is supported. >>>>>>>> For example, upon releasing 1.3, only 1.3 and 1.2 will still be >>>>>>> supported by the community for any required bug fixes. >>>>>>>> I think this was initially decided not only to ease old version >>>>>>> maintenance efforts for the community, but also as a means to let >> users >>>>>>> upgrade their Flink versions in a reasonable pace (at least every >> other >>>>>>> major release.) >>>>>>>> >>>>>>>> Therefore, I think its also reasonable to also clearly state that >>>>>>> savepoints compatibility will only be guaranteed for the previous >>>>> release. >>>>>>>> Although I think at the moment almost, if not all, of the current >> code >>>>>>> still maintains compatibility for 1.1, in the long run these >> migration >>>>>>> codes would definitely start to pile up and pollute the actual >> codebase >>>>> if >>>>>>> we try to always be compatible with all previous versions. >>>>>>>> >>>>>>>> Cheers, >>>>>>>> Gordon >>>>>>>> >>>>>>>> >>>>>>>> On 21 May 2017 at 2:24:53 AM, Kostas Kloudas ( >>>>>>> k.klou...@data-artisans.com) wrote: >>>>>>>> >>>>>>>> Hi Chesnay, >>>>>>>> >>>>>>>> I believe that for APIs we already have a pretty clear policy with >> the >>>>>>> annotations. >>>>>>>> I was referring to savepoints and state related backwards >>>>> compatibility. >>>>>>>> >>>>>>>> >>>>>>>>> On May 20, 2017, at 7:20 PM, Chesnay Schepler <ches...@apache.org> >>>>>>> wrote: >>>>>>>>> >>>>>>>>> I think it would be a good to clarify what kind of >>>>>>> backwards-compatibilitiy we're talking about here. As in are we >> talking >>>>>>> about APIs or savepoints? >>>>>>>>> >>>>>>>>> On 20.05.2017 19:09, Kostas Kloudas wrote: >>>>>>>>>> Hi all, >>>>>>>>>> >>>>>>>>>> As we are getting closer to releasing Flink-1.3, I would like to >> open >>>>>>> a discussion >>>>>>>>>> on how far back we provide backwards compatibility for. >>>>>>>>>> >>>>>>>>>> The reason for opening the discussion is that i) for the users and >>>>> for >>>>>>> the >>>>>>>>>> adoption of the project, it is good to have an explicitely stated >>>>>>> policy that implies >>>>>>>>>> certain guarantees, and ii) keeping code and tests for backwards >>>>>>> compatibility with >>>>>>>>>> Flink-1.1 does not offer much. On the contrary, I think that it >> leads >>>>>>> to: >>>>>>>>>> >>>>>>>>>> 1) dead or ugly code in the codebase, e.g. deprecated class fields >>>>>>> that could go away and >>>>>>>>>> ugly if() loops (see aligned window operators that were >> deprecated in >>>>>>> 1.2 and are now >>>>>>>>>> normal windows), etc >>>>>>>>>> 2) expensive tests (as, normally, they read from a savepoint) >>>>>>>>>> 3) binary files in the codebase for holding the aforementioned >>>>>>> savepoints >>>>>>>>>> >>>>>>>>>> My proposal for such a policy would be to offer backwards >>>>>>> compatibility for one previous version. >>>>>>>>>> >>>>>>>>>> This means that 1.3 will be compatible with 1.2 (not 1.1). This >> still >>>>>>> allows a clear >>>>>>>>>> "backwards compatibility" path when jumping versions (a user that >>>>> goes >>>>>>>>>> from 1.1 to 1.3 can go initially 1.1 -> 1.2, take a savepoint, and >>>>>>> then 1.2 -> 1.3), >>>>>>>>>> while also allowing us to clean up the codebase a bit. >>>>>>>>>> >>>>>>>>>> What do you think? >>>>>>>>>> >>>>>>>>>> Kostas >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>> >>>>> >>> >> >>
