Thank you for starting this discussion Timo! Maybe we should draft a Wiki page outlining our deprecation and removal policies for legacy code in Flink. We'll need such a document anyways, and summarizing the consensus from this discussion into a document would allow us to move forward with this discussion.
Once we agree on a policy, we can do the required code changes (deprecation interface), update the release manager guidelines etc. Timo, do you want to drive this? On Thu, Jan 21, 2021 at 3:37 PM Matthias Pohl <matth...@ververica.com> wrote: > > > > I would prefer not to rely on the Jira for marking when something is > > supposed to be deleted. If `@Deprecated(since, planned_to_remove_on)` > would > > have two obligatory parameters, there would be no way to "forget" about > > marking it and it would be also self documenting (I don't imagine users > > using JIRA to check this kind of things). We can have Jira tickets for > > those things for tracking purposes on the JIRA release board, but relying > > only on JIRA tickets I think is just asking for inconsistencies. > > > Yes, I totally agree with you. The annotation should be the way to document > it. I just saw the Jira issue proposal as a good way to track these things > within the team for code that is not visited regularly. But as I said, that > depends on how each team organizes itself. I don't see this as a process > everyone needs to follow. > > I agree it would depend on the feature, hence different features might have > > longer or shorter "unstable" timeframes. But I'm afraid if we won't start > > thinking about fixing this timeframe, we would too often end up with > > perpetually "unstable" APIs. I don't know where I would draw the line > > exactly, but assuming we want to have stable APIs, if something is marked > > `@PublicEvolving` or `@Experimental` for 3 years, IMO it should be > switched > > to `@Public` by default (or be moved out of the main repo?). > > > We could use the timestamp for the @PublicEvolving and @Experimental as an > indicator to revisit the annotation. Let's say, we revisit each annotation > after a specific amount of time (e.g. after two releases) to discuss > whether the API is still evolving or actually stable. If it's stable, we > change the annotation to @Public. If it's not stable, we update the > timestamp. > > Best, > Matthias > > On Wed, Jan 20, 2021 at 2:01 PM Piotr Nowojski <pnowoj...@apache.org> > wrote: > > > Hi, > > > > I would prefer not to rely on the Jira for marking when something is > > supposed to be deleted. If `@Deprecated(since, planned_to_remove_on)` > would > > have two obligatory parameters, there would be no way to "forget" about > > marking it and it would be also self documenting (I don't imagine users > > using JIRA to check this kind of things). We can have Jira tickets for > > those things for tracking purposes on the JIRA release board, but relying > > only on JIRA tickets I think is just asking for inconsistencies. > > > > > Is it actually possible to have a fixed timeframe for these annotations > > to change? > > > I would imagine that it depends on the underlying feature how long an > > API is @PublicEvolving or @Experimental? > > > > I agree it would depend on the feature, hence different features might > > have longer or shorter "unstable" timeframes. But I'm afraid if we won't > > start thinking about fixing this timeframe, we would too often end up > with > > perpetually "unstable" APIs. I don't know where I would draw the line > > exactly, but assuming we want to have stable APIs, if something is marked > > `@PublicEvolving` or `@Experimental` for 3 years, IMO it should be > switched > > to `@Public` by default (or be moved out of the main repo?). > > > > Piotrek > > > > śr., 20 sty 2021 o 09:54 Matthias Pohl <matth...@ververica.com> > > napisał(a): > > > >> Thanks Timo for opening this discussion. > >> > >> +1 I like the idea of adding a deprecation deadline and/or information > >> when > >> the > >> functionality was deprecated. It looks like this is already done in the > >> PyFlink code. > >> > >> Creating a JIRA issue for removing the functionality, as Till suggested, > >> might help to > >> maintain this process of removing the deprecated functionality. I'd > prefer > >> that over > >> relying on the release manager (assuming that he/she would run the check > >> as > >> part > >> of the release process) to identify functionality that should have been > >> removed as > >> part of the release. But ok, that might be a team decision. > >> > >> For the connectors: Can't we assume that users would reach out to us if > we > >> deprecate > >> a connector assuming that they can conclude that this connector will, > >> otherwise, disappear. > >> Maybe, that needs to be mentioned in the deprecation information as > well, > >> then. > >> This would have the benefit of getting direct feedback about how much a > >> connector is still in > >> use and may open the doors for other contributors to offer help like it > >> happened for the > >> Mesos support [1]. > >> > >> And about the idea of adding such deadlines to @Public, @PublicEvolving, > >> and @Experimental: > >> Is it actually possible to have a fixed timeframe for these annotations > to > >> change? I would > >> imagine that it depends on the underlying feature how long an API > >> is @PublicEvolving or > >> @Experimental? But it sounds still like a good idea to trigger warnings > >> for > >> those annotations > >> in case they haven't been touched for a while. Therefore, I would second > >> this suggestion. > >> > >> Best, > >> Matthias > >> > >> [1] > >> > >> > http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/SURVEY-Remove-Mesos-support-td45974.html#a45985 > >> > >> On Tue, Jan 19, 2021 at 10:15 AM Till Rohrmann <trohrm...@apache.org> > >> wrote: > >> > >> > Thanks a lot for starting this discussion Timo. I like the idea of > >> setting > >> > more explicit guidelines for deprecating functionality. > >> > > >> > I really like the idea of adding with the @Deprecated annotation since > >> when > >> > the function is deprecated. Based on that one can simply search for > >> > features which should be removed in a given release. Alternatively, > one > >> > could as you said also state the removal version. > >> > > >> > I think what also works is to directly create a critical JIRA issue > with > >> > removing functionality as soon as one deprecates something. The > problem > >> was > >> > often that after deprecating something, it gets forgotten. > >> > > >> > For dropping connectors I am a bit uncertain. From a project > management > >> > perspective it sounds like a good idea to not have to support > connectors > >> > which are no longer supported for some time. However, what if this > >> > connector is still very popular and in heavy use by our users? Just > >> because > >> > an external system or a version of it is no longer maintained does not > >> mean > >> > that the system is no longer used. I think our current practice with > >> trying > >> > to judge whether our users still use this feature/connector works > >> somewhat. > >> > On the other hand, having these guidelines would probably make it > >> easier to > >> > argue to remove something even if there are still a couple of users. > >> > > >> > Cheers, > >> > Till > >> > > >> > On Mon, Jan 18, 2021 at 11:37 AM Yun Gao <yungao...@aliyun.com.invalid > > > >> > wrote: > >> > > >> > > Hi, > >> > > > >> > > Very thanks for @Timo to initiate the discussion! > >> > > > >> > > I would also +1 for providing some informations to users via > >> annotations > >> > > or documents in advanced to not suprise users before we actually > >> remove > >> > > the legacy code. > >> > > If we finally decide to change one functionality that user could > >> sense, > >> > > perhaps one > >> > > premise is that Flink has provided a replacement for that one and > >> users > >> > > could transfer their > >> > > applications easily. Then we might also consider have one dedicated > >> > > documentation page > >> > > to list the functionalities to change and how users could do the > >> > transfer. > >> > > > >> > > To make the decision of whether to remove some legacy code, we might > >> also > >> > > consider to have a survey > >> > > like the one we did for mesos support [1] to see how this > >> functionality > >> > is > >> > > used. > >> > > > >> > > Best, > >> > > Yun > >> > > > >> > > > >> > > [1] > >> > > > >> > > >> > https://lists.apache.org/thread.html/r139b11190a6d1f09c9e44d5fa985fd8d310347e66d2324ec1f0c2d87%40%3Cuser.flink.apache.org%3E > >> > > > >> > > > >> > > > >> > > ------------------Original Mail ------------------ > >> > > Sender:Piotr Nowojski <pnowoj...@apache.org> > >> > > Send Date:Mon Jan 18 18:23:36 2021 > >> > > Recipients:dev <dev@flink.apache.org> > >> > > Subject:Re: [DISCUSS] Dealing with deprecated and legacy code in > Flink > >> > > Hi Timo, > >> > > > >> > > Thanks for starting this discussion. I'm not sure how we should > >> approach > >> > > this topic and what should be our final recommendation, but > definitely > >> > > clearing up a couple of things would be helpful. > >> > > > >> > > For starters, I agree it would be good to have some more > information, > >> > > besides just "@Deprecated" annotations. Is it possible to extend > >> > > annotations with informations like: > >> > > - from which version was it deprecated > >> > > - when is it planned to be removed (we could always mark `2.0` as > >> "never" > >> > > ;) ) > >> > > - add maybe some pre/post release step of verifying that removal has > >> > > actually happened? > >> > > > >> > > ? > >> > > > >> > > On the other hand, I think it's very important to maintain backward > >> > > compatibility with Flink as much as possible. As a developer I don't > >> > > like dealing with this, but as a user I hate dealing with > incompatible > >> > > upgrades even more. So all in all, I would be in favour of putting > >> more > >> > > effort not in deprecating and removing APIs, but making sure that > they > >> > are > >> > > stable. > >> > > > >> > > Stephan Ewan also raised a point sometime ago, that in the recent > >> past, > >> > we > >> > > developed a habit of marking everything as `@Experimental` or > >> > > `@PublicEvolving` and leaving it as that forever. Maybe we should > also > >> > > include deadlines (2 releases since introduction?) for changing > >> > > `@Experimental`/`@PublicEvolving` into `@Public` in this kind of > >> > > guidelines/automated checks? > >> > > > >> > > Piotrek > >> > > > >> > > pt., 15 sty 2021 o 13:56 Timo Walther <twal...@apache.org> > >> napisał(a): > >> > > > >> > > > Hi everyone, > >> > > > > >> > > > I would like to start a discussion how we treat deprecated and > >> legacy > >> > > > code in Flink in the future. During the last years, our code base > >> has > >> > > > grown quite a bit and a couple of interfaces and components have > >> been > >> > > > reworked on the way. > >> > > > > >> > > > I'm sure each component has a few legacy parts that are waiting > for > >> > > > removal. Apart from keeping outdated API around for a couple of > >> > releases > >> > > > until users have updated their code, it is also often easier to > just > >> > put > >> > > > a @Deprecation annotation and postpone the actual change. > >> > > > > >> > > > When looking at the current code, we have duplicated SQL planners, > >> > > > duplicated APIs (DataSet/DataStream), duplicated source/sink > >> > interfaces, > >> > > > outdated connectors (Elasticsearch 5?) and dependencies (Scala > >> 2.11?). > >> > > > > >> > > > I'm wondering whether we should come up with some > legacy/deprecation > >> > > > guidelines for the future. > >> > > > > >> > > > Some examples: > >> > > > > >> > > > - I could imagine new Flink-specific annotations for documenting > (in > >> > > > code) in which version an interface was deprecated and when the > >> planned > >> > > > removal should take place. > >> > > > - Or guidelines that we drop a connector when the external project > >> does > >> > > > not maintain the version for 6 months etc. > >> > > > > >> > > > Plannable removal dates should also help users to not be surprised > >> when > >> > > > a connector or Scala version is not supported anymore. > >> > > > > >> > > > What do you think? I'm very happy to hear more opinions. > >> > > > > >> > > > Regards, > >> > > > Timo > >> > > > > >> > > > > >> > > > > >> > > > > >> > > > > >> > > > > >> > > > >> > > >
