For many parts of the code, I would agree with Aljoscha. However, I can also
see notable exceptions, such as maintaining support for the legacy state from
Flink <=1.1. For example, I think dropping support for this can simplify new
developments such as fast local recovery or state replication quiet a bit
because this is a special case that runs through a lot of code from backend to
JM. So besides this general discussion about a backwards compatible policy, do
you think it could make sense to start another concrete discussion about if we
still must or want backwards compatibility to Flink 1.1 in Flink 1.4?
> Am 29.05.2017 um 12:08 schrieb Aljoscha Krettek <aljos...@apache.org>:
>
> Normally, I’m the first one to suggest removing everything that is not
> absolutely necessary in order to have a clean code base. On this issue,
> though, I think we should support restoring from old Savepoints as far back
> as possible if it does not make the code completely unmaintainable. Some
> users might jump versions and always forcing them to go though every version
> from their old version to the current version doesn’t seem feasible and might
> put off some users.
>
> So far, I think the burden of supporting restore from 1.1 is still small
> enough and with each new version the changes between versions become less and
> less. The changes from 1.2 to the upcoming 1.3 are quite minimal, I think.
>
> Best,
> Aljoscha
>> On 24. May 2017, at 17:58, Ted Yu <yuzhih...@gmail.com> wrote:
>>
>> bq. about having LTS versions once a year
>>
>> +1 to the above.
>>
>> There may be various reasons users don't want to upgrade (after new
>> releases come out). We should give such users enough flexibility on the
>> upgrade path.
>>
>> Cheers
>>
>> On Wed, May 24, 2017 at 8:39 AM, Kostas Kloudas <k.klou...@data-artisans.com
>>> wrote:
>>
>>> Hi all,
>>>
>>> For the proposal of having a third party tool, I agree with Ted.
>>> Maintaining
>>> it is a big and far from trivial effort.
>>>
>>> Now for the window of backwards compatibility, I would argue that even if
>>> for some users 4 months (1 release) is not enough to bump their Flink
>>> version,
>>> the proposed policy guarantees that there will always be a path from any
>>> old
>>> version to any subsequent one.
>>>
>>> Finally, for the proposal about having LTS versions once a year, I am not
>>> sure if this will reduce or create more overhead. If I understand the plan
>>> correctly, this would mean that the community will have to maintain
>>> 2 or 3 LTS versions and the last two major ones, right?
>>>
>>>> On May 22, 2017, at 7:31 PM, Ted Yu <yuzhih...@gmail.com> wrote:
>>>>
>>>> For #2, it is difficult to achieve:
>>>>
>>>> a. maintaining savepoint migration is non-trivial and should be reviewed
>>> by
>>>> domain experts
>>>> b. how to certify such third-party tool
>>>>
>>>> Cheers
>>>>
>>>> On Mon, May 22, 2017 at 3:04 AM, 施晓罡 <shixiaoga...@gmail.com> wrote:
>>>>
>>>>> Hi all,
>>>>>
>>>>> Currently, we work a lot in the maintenance of compatibility.
>>>>> There exist much code in runtime to support the migration of savepoints
>>>>> (most of which are deprecated), making it hard to focus on the current
>>>>> implementation.
>>>>> When more versions are released, much more efforts will be needed if we
>>>>> try to make these released versions compatible.
>>>>>
>>>>> I agree with Tzu-Li that we should provide a method to let users upgrade
>>>>> Flink in a reasonable pace.
>>>>> But i am against the proposal that we only offer backwards compatibility
>>>>> for one previous version.
>>>>> According our time-based release model, a major version is released
>>> every
>>>>> four month.
>>>>> That means, users have to upgrade their versions every 8 months.
>>> Otherwise
>>>>> they will have difficulties in the migration of existing savepoints.
>>>>>
>>>>> My suggestions include
>>>>>
>>>>> (1) We can release Long-Term Support (LTS) versions which are widely
>>>>> adopted in other open-source projects.
>>>>> LTS versions should be stable and are free of found bugs. Savepoints in
>>>>> LTS versions are guaranteed to be back-compatible so that users can
>>> easily
>>>>> upgrade to newer LTS versions.
>>>>>
>>>>> The releasing of LTS versions is slower than that of major versions
>>> (maybe
>>>>> once a year, determined by users’ upgrade frequency).
>>>>> Each LTS version will be supported a period of time and typically there
>>>>> are no more than three active LTS versions.
>>>>> By encouraging users to use LTS versions, we can ease the maintenance of
>>>>> released versions (bug fixes, back compatibility, and critical
>>> performance
>>>>> improvement).
>>>>>
>>>>> (2) We can provide a third-party tool to do the migration of
>>> old-versioned
>>>>> savepoints.
>>>>> When users upgrade their versions, they can use the provided tool to
>>>>> migrate existing savepoints.
>>>>> This can help move the code for savepoint migration out of the actual
>>>>> codebase, making code focuses on current implementation.
>>>>>
>>>>> What do you think?
>>>>>
>>>>> Regards,
>>>>> Xiaogang
>>>>>
>>>>>
>>>>>> 在 2017年5月22日，下午1:39，Tzu-Li (Gordon) Tai <tzuli...@apache.org> 写道：
>>>>>>
>>>>>> Hi Kostas,
>>>>>>
>>>>>> Thanks for bringing this up!
>>>>>> I think it is reasonable to keep this coherent with our timely-based
>>>>> release model guarantees.
>>>>>>
>>>>>> With the timely-based release model, there is a guarantee that the
>>>>> current latest major version and the previous one is supported.
>>>>>> For example, upon releasing 1.3, only 1.3 and 1.2 will still be
>>>>> supported by the community for any required bug fixes.
>>>>>> I think this was initially decided not only to ease old version
>>>>> maintenance efforts for the community, but also as a means to let users
>>>>> upgrade their Flink versions in a reasonable pace (at least every other
>>>>> major release.)
>>>>>>
>>>>>> Therefore, I think its also reasonable to also clearly state that
>>>>> savepoints compatibility will only be guaranteed for the previous
>>> release.
>>>>>> Although I think at the moment almost, if not all, of the current code
>>>>> still maintains compatibility for 1.1, in the long run these migration
>>>>> codes would definitely start to pile up and pollute the actual codebase
>>> if
>>>>> we try to always be compatible with all previous versions.
>>>>>>
>>>>>> Cheers,
>>>>>> Gordon
>>>>>>
>>>>>>
>>>>>> On 21 May 2017 at 2:24:53 AM, Kostas Kloudas (
>>>>> k.klou...@data-artisans.com) wrote:
>>>>>>
>>>>>> Hi Chesnay,
>>>>>>
>>>>>> I believe that for APIs we already have a pretty clear policy with the
>>>>> annotations.
>>>>>> I was referring to savepoints and state related backwards
>>> compatibility.
>>>>>>
>>>>>>
>>>>>>> On May 20, 2017, at 7:20 PM, Chesnay Schepler <ches...@apache.org>
>>>>> wrote:
>>>>>>>
>>>>>>> I think it would be a good to clarify what kind of
>>>>> backwards-compatibilitiy we're talking about here. As in are we talking
>>>>> about APIs or savepoints?
>>>>>>>
>>>>>>> On 20.05.2017 19:09, Kostas Kloudas wrote:
>>>>>>>> Hi all,
>>>>>>>>
>>>>>>>> As we are getting closer to releasing Flink-1.3, I would like to open
>>>>> a discussion
>>>>>>>> on how far back we provide backwards compatibility for.
>>>>>>>>
>>>>>>>> The reason for opening the discussion is that i) for the users and
>>> for
>>>>> the
>>>>>>>> adoption of the project, it is good to have an explicitely stated
>>>>> policy that implies
>>>>>>>> certain guarantees, and ii) keeping code and tests for backwards
>>>>> compatibility with
>>>>>>>> Flink-1.1 does not offer much. On the contrary, I think that it leads
>>>>> to:
>>>>>>>>
>>>>>>>> 1) dead or ugly code in the codebase, e.g. deprecated class fields
>>>>> that could go away and
>>>>>>>> ugly if() loops (see aligned window operators that were deprecated in
>>>>> 1.2 and are now
>>>>>>>> normal windows), etc
>>>>>>>> 2) expensive tests (as, normally, they read from a savepoint)
>>>>>>>> 3) binary files in the codebase for holding the aforementioned
>>>>> savepoints
>>>>>>>>
>>>>>>>> My proposal for such a policy would be to offer backwards
>>>>> compatibility for one previous version.
>>>>>>>>
>>>>>>>> This means that 1.3 will be compatible with 1.2 (not 1.1). This still
>>>>> allows a clear
>>>>>>>> "backwards compatibility" path when jumping versions (a user that
>>> goes
>>>>>>>> from 1.1 to 1.3 can go initially 1.1 -> 1.2, take a savepoint, and
>>>>> then 1.2 -> 1.3),
>>>>>>>> while also allowing us to clean up the codebase a bit.
>>>>>>>>
>>>>>>>> What do you think?
>>>>>>>>
>>>>>>>> Kostas
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>
>>>
>
