Thanks for the beautiful sheets, Jane. 1. This sheet < > https://docs.google.com/spreadsheets/d/1dZBNHLuAHYJt3pFU8ZtfUzrYyf2ZFQ6wybDXGS1bHno/edit?usp=sharing> > summarizes the user-facing classes and methods that need to be deprecated > under the flink-table module, some of which are marked with a red > background and belong to the APIs that need to be depreciated but are not > explicitly marked in the code. This mainly includes legacy table > source/sink, legacy table schema, legacy SQL function, and some internal > APIs designed for Paimon but are now obsolete. > IIUC, you are suggesting to mark the classes with a red background as `@Deprecated` in 1.18?
- +1 for deprecating `StreamRecordTimestamp` & `ExistingField` in 1.18. Based on your description, it seems these were not marked by mistake. Let's fix them. - For the ManagedTable related classes, is there any FLIP explicitly that decides to deprecate them? If not, I think it would be nice to have one, to formally decide the deprecation with a vote. I'd expect such a FLIP can be quite lightweight. - For `SourceFunctionProvider`, please beware there are recently some objections against deprecating `SourceFunction`, and the code changes marking it as `@Deprecated` might be reverted. See [1][2] for more details. So my question is, if `SourceFunction` will not be deprecated until all sub-tasks in FLINK-28045 are resolved, do you still think we should deprecate `SourceFunctionProvider` now? 2. In addition, during the process of organizing, it was found that some > APIs under the flink-table-api-java and flink-table-common modules do not > have an explicit API annotation (you can find detailed information in this > sheet < > https://docs.google.com/spreadsheets/d/1e8M0tUtKkZXEd8rCZtZ0C6Ty9QkNaPySsrCgz0vEID4/edit?usp=sharing>). > I suggest explicitly marking the level for these APIs. I'm in general +1 to add the missing API annotations. However, I don't have the expertise to comment on the classes and suggestd API levels being listed. 3. As there are still some internal and test code dependencies on these > APIs, can we first gradually migrate these dependencies to alternative > APIs to make the deprecation process relatively easy? > That makes sense to me. I think the purpose of trying to mark the APIs as deprecated in 1.18 is to send users the signal early that these APIs will be removed. As for the internal and test code dependencies, I don't see any problem in gradually migrating them. Best, Xintong [1] https://lists.apache.org/thread/734zhkvs59w2o4d1rsnozr1bfqlr6rgm [2] https://issues.apache.org/jira/browse/FLINK-28046 On Wed, Jul 19, 2023 at 11:41 AM Jane Chan <qingyue....@gmail.com> wrote: > Hi Xintong, > > Thanks for driving this topic. Regarding the Table API deprecation, I can > provide some details to help with the process. > > 1. This sheet > < > https://docs.google.com/spreadsheets/d/1dZBNHLuAHYJt3pFU8ZtfUzrYyf2ZFQ6wybDXGS1bHno/edit?usp=sharing > > > summarizes > the user-facing classes and methods that need to be deprecated under the > flink-table module, some of which are marked with a red background and > belong to the APIs that need to be depreciated but are not explicitly > marked in the code. This mainly includes legacy table source/sink, > legacy > table schema, legacy SQL function, and some internal APIs designed for > Paimon but are now obsolete. > 2. In addition, during the process of organizing, it was found that some > APIs under the flink-table-api-java and flink-table-common modules do > not > have an explicit API annotation (you can find detailed information in > this > sheet > < > https://docs.google.com/spreadsheets/d/1e8M0tUtKkZXEd8rCZtZ0C6Ty9QkNaPySsrCgz0vEID4/edit?usp=sharing > >). > I suggest explicitly marking the level for these APIs. > 3. As there are still some internal and test code dependencies on these > APIs, can we first gradually migrate these dependencies to alternative > APIs to make the deprecation process relatively easy? > > I hope this helps. > > Best, > Jane > > On Tue, Jul 11, 2023 at 12:08 PM Xintong Song <tonysong...@gmail.com> > wrote: > > > Thanks for bringing this up, Matthias. I've replied to you in the other > > thread[1]. > > > > Best, > > > > Xintong > > > > > > [1] https://lists.apache.org/thread/l3dkdypyrovd3txzodn07lgdwtwvhgk4 > > > > On Mon, Jul 10, 2023 at 8:37 PM Matthias Pohl > > <matthias.p...@aiven.io.invalid> wrote: > > > > > @Xintong did you come across FLINK-3957 [1] when looking for deprecated > > > issues? There are a few subtasks that would require deprecations as > well > > as > > > far as I can see. For some I'm wondering whether we should add them to > > the > > > release 2.0 feature list [2] in some way and (as a consequence) missed > > them > > > in FLINK-32557 [3], e.g.: > > > - FLINK-4675 [4] > > > - FLINK-14068 [5] (listed in the release 2.0 feature list [2] but not > > > listed in FLINK-32557 [3]) > > > - FLINK-5126 [6] > > > - FLINK-13926 [7] > > > > > > For some other subtasks, I'm not 100% sure whether they still apply. > > Others > > > might resolve by removing the DataSet or Scala API. But for the 4 > > mentioned > > > above we might need to deprecate API in 1.18. > > > > > > Matthias > > > > > > [1] https://issues.apache.org/jira/browse/FLINK-3957 > > > [2] https://cwiki.apache.org/confluence/display/FLINK/2.0+Release > > > [3] https://issues.apache.org/jira/browse/FLINK-32557 > > > > > > [4] https://issues.apache.org/jira/browse/FLINK-4675 > > > [5] https://issues.apache.org/jira/browse/FLINK-14068 > > > [6] https://issues.apache.org/jira/browse/FLINK-5126 > > > [7] https://issues.apache.org/jira/browse/FLINK-13926 > > > > > > On Fri, Jul 7, 2023 at 12:59 PM Jing Ge <j...@ververica.com.invalid> > > > wrote: > > > > > > > Hi Alex, > > > > > > > > I would follow FLIP-197 and try to release them asap depending on dev > > > > resources and how difficult those issues are. The fastest timeline is > > the > > > > period defined in FLIP-197 in ideal conditions. > > > > > > > > Best regards, > > > > Jing > > > > > > > > On Fri, Jul 7, 2023 at 12:20 PM Alexander Fedulov < > > > > alexander.fedu...@gmail.com> wrote: > > > > > > > > > @Xintong > > > > > > - IIUC, the testing scenario you described is like blocking the > > > source > > > > > for > > > > > > proceeding (emit data, finish, etc.) until a checkpoint is > > finished. > > > > > > > > > > It is more tricky than that - we need to prevent the Sink from > > > receiving > > > > a > > > > > checkpoint barrier until the Source is done emitting a given set of > > > > > records. In > > > > > the current tests, which are also used for V2 Sinks, SourceFunction > > > > > controls > > > > > when the Sink is "allowed" to commit by holding the checkpoint lock > > > while > > > > > producing the records. The lock is not available in the new Source > by > > > > > design > > > > > and we need a solution that provides the same functionality > (without > > > > > modifying > > > > > the Sinks). I am currently checking if a workaround is at all > > possible > > > > > without > > > > > adjusting anything in the Source interface. > > > > > > > > > > > I may not have understood all the details, but based on what you > > > > > described > > > > > > I'd hesitate to block the deprecation / removal of SourceFunction > > on > > > > > this. > > > > > > > > > > I don't think we should, just wanted to highlight that there are > some > > > > > unknowns > > > > > with respect to estimating the amount of work required. > > > > > > > > > > @Jing > > > > > I want to understand in which release would you target graduation > of > > > the > > > > > mentioned connectors to @Public/@PublicEvolving - basically the > > > > anticipated > > > > > timeline of the steps in both options with respect to releases. > > > > > > > > > > Best, > > > > > Alex > > > > > > > > > > On Fri, 7 Jul 2023 at 10:53, Xintong Song <tonysong...@gmail.com> > > > wrote: > > > > > > > > > > > Thanks all for the discussion. I've created FLINK-32557 for this. > > > > > > > > > > > > Best, > > > > > > > > > > > > Xintong > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Jul 6, 2023 at 1:00 AM Jing Ge > <j...@ververica.com.invalid > > > > > > > > wrote: > > > > > > > > > > > > > Hi Alex, > > > > > > > > > > > > > > > > > > > > > > > 3. remove SinkFunction. > > > > > > > > Which steps do you imply for the 1.18 release and for the 2.0 > > > > > release? > > > > > > > > > > > > > > > > > > > > > > for 2.0 release. 1.18 will be released soon. > > > > > > > > > > > > > > Best regards, > > > > > > > Jing > > > > > > > > > > > > > > > > > > > > > On Wed, Jul 5, 2023 at 1:08 PM Alexander Fedulov < > > > > > > > alexander.fedu...@gmail.com> wrote: > > > > > > > > > > > > > > > @Jing > > > > > > > > Just to clarify, when you say: > > > > > > > > > > > > > > > > 3. remove SinkFunction. > > > > > > > > Which steps do you imply for the 1.18 release and for the 2.0 > > > > > release? > > > > > > > > > > > > > > @Xintong > > > > > > > > A side note - with the new Source API we lose the ability to > > > > control > > > > > > > > checkpointing from the source since there is no lock anymore. > > > This > > > > > > > > functionality > > > > > > > > is currently used in a variety of tests for the Sinks - the > > tests > > > > > that > > > > > > > rely > > > > > > > > on tight > > > > > > > > synchronization between specific elements passed from the > > source > > > > to > > > > > > the > > > > > > > > sink before > > > > > > > > allowing a checkpoint to complete (see FiniteTestSource [1]). > > > Since > > > > > > > FLIP-27 > > > > > > > > Sources rely > > > > > > > > on decoupling via the mailbox, without exposing the lock, it > is > > > not > > > > > > > > immediately clear > > > > > > > > if it is possible to achieve the same functionality without > > major > > > > > > > > extensions in the > > > > > > > > runtime for such testing purposes. My hope initially was that > > > only > > > > > the > > > > > > > > legacy Sinks > > > > > > > > relied on this - this would have made it possible to drop > > > > > > > > SourceFunction+SinkFunction > > > > > > > > together, but, in fact, it also already became part of the > new > > > > SinkV2 > > > > > > > > testing IT suits > > > > > > > > [2]. Moreover, I know of at least one major connector that > also > > > > > relies > > > > > > on > > > > > > > > it for > > > > > > > > verifying committed sink metadata for a specific set of > records > > > > > > (Iceberg) > > > > > > > > [3]. In my > > > > > > > > estimation this currently presents a major blocker for the > > > > > > SourceFunction > > > > > > > > removal. > > > > > > > > > > > > > > > > [1] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/apache/flink/blob/master/flink-test-utils-parent/flink-test-utils/src/main/java/org/apache/flink/streaming/util/FiniteTestSource.java > > > > > > > > [2] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-files/src/test/java/org/apache/flink/connector/file/sink/StreamingExecutionFileSinkITCase.java#L132 > > > > > > > > [3] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/apache/iceberg/blob/master/flink/v1.17/flink/src/test/java/org/apache/iceberg/flink/source/BoundedTestSource.java#L75C1-L85C2 > > > > > > > > > > > > > > > > Best, > > > > > > > > Alex > > > > > > > > > > > > > > > > On Wed, 5 Jul 2023 at 10:47, Chesnay Schepler < > > > ches...@apache.org> > > > > > > > wrote: > > > > > > > > > > > > > > > > > There's a whole bunch of metric APIs that would need to be > > > > > > deprecated. > > > > > > > > > That is of course if the metric FLIPs are being accepted. > > > > > > > > > > > > > > > > > > Which makes me wonder if we aren't doing things the wrong > way > > > > > around; > > > > > > > > > shouldn't the decision to deprecate an API be part of the > > FLIP > > > > > > > > discussion? > > > > > > > > > > > > > > > > > > On 05/07/2023 07:39, Xintong Song wrote: > > > > > > > > > > Thanks all for the discussion. > > > > > > > > > > > > > > > > > > > > It seems to me there's a consensus on marking the > following > > > as > > > > > > > > deprecated > > > > > > > > > > in 1.18: > > > > > > > > > > - DataSet API > > > > > > > > > > - SourceFunction > > > > > > > > > > - Queryable State > > > > > > > > > > - All Scala APIs > > > > > > > > > > > > > > > > > > > > More time is needed for deprecating SinkFunction. > > > > > > > > > > > > > > > > > > > > I'll leave this discussion open for a few more days. And > if > > > > > there's > > > > > > > no > > > > > > > > > > objections, I'll create JIRA tickets accordingly. > > > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > > > > > > > > > > > > Xintong > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Jul 5, 2023 at 1:34 PM Xintong Song < > > > > > tonysong...@gmail.com > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > >> Thanks for the input, Jing. I'd also be +1 for option 1. > > > > > > > > > >> > > > > > > > > > >> Best, > > > > > > > > > >> > > > > > > > > > >> Xintong > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > >> On Mon, Jul 3, 2023 at 7:20 PM Jing Ge > > > > > <j...@ververica.com.invalid > > > > > > > > > > > > > > > > wrote: > > > > > > > > > >> > > > > > > > > > >>> Hi Xingtong, > > > > > > > > > >>> > > > > > > > > > >>> Option 1, secure plan would be: > > > > > > > > > >>> > > > > > > > > > >>> 1. graduate kafka, File, JDBC connectors to @Public > > > > > > > > > >>> 2. graduate SinkV2 to @Public > > > > > > > > > >>> 3. remove SinkFunction. > > > > > > > > > >>> > > > > > > > > > >>> Option 2, risky plan but at a fast pace: > > > > > > > > > >>> > > > > > > > > > >>> 1. graduate SinkV2 to @Public and expecting more > > > maintenance > > > > > > effort > > > > > > > > > since > > > > > > > > > >>> there are many known and unsolved issues. > > > > > > > > > >>> 2. remove SinkFunction. > > > > > > > > > >>> 3. It depends on the connectors' contributors whether > > > > > connectors > > > > > > > can > > > > > > > > > >>> upgrade to Flink 2.0, since we moved forward with > SinkV2 > > > API > > > > > > > without > > > > > > > > > >>> taking > > > > > > > > > >>> care of implementations in external connectors. > > > > > > > > > >>> > > > > > > > > > >>> I am ok with both of them and personally prefer option > 1. > > > > > > > > > >>> > > > > > > > > > >>> Best regards, > > > > > > > > > >>> Jing > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > > >>> On Fri, Jun 30, 2023 at 3:41 AM Xintong Song < > > > > > > > tonysong...@gmail.com> > > > > > > > > > >>> wrote: > > > > > > > > > >>> > > > > > > > > > >>>> I see. Thanks for the explanation. I may have not > looked > > > > into > > > > > > this > > > > > > > > > >>> deeply > > > > > > > > > >>>> enough, and would trust the decision from you and the > > > > > community > > > > > > > > > members > > > > > > > > > >>> who > > > > > > > > > >>>> participated in the discussion & vote. > > > > > > > > > >>>> > > > > > > > > > >>>> Best, > > > > > > > > > >>>> > > > > > > > > > >>>> Xintong > > > > > > > > > >>>> > > > > > > > > > >>>> > > > > > > > > > >>>> > > > > > > > > > >>>> On Thu, Jun 29, 2023 at 10:28 PM Alexander Fedulov < > > > > > > > > > >>>> alexander.fedu...@gmail.com> wrote: > > > > > > > > > >>>> > > > > > > > > > >>>>>> However, I'm not sure about 2. > > > > > > > > > >>>>> I am not aware of a bylaw that states the specific > > > > > requirements > > > > > > > in > > > > > > > > > >>> order > > > > > > > > > >>>> to > > > > > > > > > >>>>> mark something as @Deprecated. My understanding from > > the > > > > > > > discussion > > > > > > > > > >>> and > > > > > > > > > >>>> the > > > > > > > > > >>>>> vote was that the community recognizes the necessity > to > > > > make > > > > > it > > > > > > > > > >>> explicit > > > > > > > > > >>>>> that > > > > > > > > > >>>>> the usage of the SourceFunction API is discouraged. > > This > > > > can > > > > > > > > actually > > > > > > > > > >>>>> stimulate > > > > > > > > > >>>>> authors of connectors that rely on this very specific > > and > > > > > > > > > non-baseline > > > > > > > > > >>>>> functionality to contribute extensions to the new > > Source > > > > API > > > > > > > > > >>> themselves > > > > > > > > > >>>> in > > > > > > > > > >>>>> order to > > > > > > > > > >>>>> close the gap. ExternallyInducedSource, for instance, > > was > > > > > > driven > > > > > > > by > > > > > > > > > >>>> Pravega > > > > > > > > > >>>>> to > > > > > > > > > >>>>> begin with, since it was only needed for their > purposes > > > > [1]. > > > > > We > > > > > > > are > > > > > > > > > >>> not > > > > > > > > > >>>>> removing > > > > > > > > > >>>>> anything - until 2.0 everything will continue to work > > and > > > > we > > > > > > can > > > > > > > > work > > > > > > > > > >>> on > > > > > > > > > >>>>> resolving the limitations until then, I personally > > don't > > > > see > > > > > a > > > > > > > big > > > > > > > > > >>> issue > > > > > > > > > >>>>> here. > > > > > > > > > >>>>> > > > > > > > > > >>>>>> Do you think it is feasible to resolve them by the > > > feature > > > > > > > freeze > > > > > > > > > >>> date > > > > > > > > > >>>> of > > > > > > > > > >>>>> 1.18? > > > > > > > > > >>>>> No, these are rather complex additions that would > > > probably > > > > > > > require > > > > > > > > > >>>> FLIP(s). > > > > > > > > > >>>>> [1] > > > > > > > > > >>>>> > > > > > > > > > >>>>> > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://flink.apache.org/2022/01/20/pravega-flink-connector-101/#checkpoint-integration > > > > > > > > > >>>>> On Thu, 29 Jun 2023 at 14:25, Xintong Song < > > > > > > > tonysong...@gmail.com> > > > > > > > > > >>>> wrote: > > > > > > > > > >>>>>> Thanks for the explanation, Alex. > > > > > > > > > >>>>>> > > > > > > > > > >>>>>> Not blocking the deprecation on 1 & 3 makes sense to > > me. > > > > > > > However, > > > > > > > > > >>> I'm > > > > > > > > > >>>> not > > > > > > > > > >>>>>> sure about 2. > > > > > > > > > >>>>>> > > > > > > > > > >>>>>> It sounds to me that, without FLINK-28051 & > > FLINK-28054, > > > > > some > > > > > > of > > > > > > > > the > > > > > > > > > >>>>>> connectors cannot migrate to the new Source API, or > at > > > > least > > > > > > > > further > > > > > > > > > >>>>>> investigation is needed to understand the situation. > > If > > > > this > > > > > > is > > > > > > > > the > > > > > > > > > >>>> case, > > > > > > > > > >>>>>> we probably should not deprecate the API until these > > > > issues > > > > > > are > > > > > > > > > >>>> resolved. > > > > > > > > > >>>>>> Do you think it is feasible to resolve them by the > > > feature > > > > > > > freeze > > > > > > > > > >>> date > > > > > > > > > >>>> of > > > > > > > > > >>>>>> 1.18? > > > > > > > > > >>>>>> > > > > > > > > > >>>>>> Best, > > > > > > > > > >>>>>> > > > > > > > > > >>>>>> Xintong > > > > > > > > > >>>>>> > > > > > > > > > >>>>>> > > > > > > > > > >>>>>> > > > > > > > > > >>>>>> On Thu, Jun 29, 2023 at 8:02 PM Alexander Fedulov < > > > > > > > > > >>>>>> alexander.fedu...@gmail.com> wrote: > > > > > > > > > >>>>>> > > > > > > > > > >>>>>>> @Xintong > > > > > > > > > >>>>>>> The original discussion [1] and vote [2] converged > on > > > the > > > > > > idea > > > > > > > > > >>> that > > > > > > > > > >>>> it > > > > > > > > > >>>>> is > > > > > > > > > >>>>>>> better > > > > > > > > > >>>>>>> to make it clear to the users that they should stop > > > using > > > > > > > > > >>>>> SourceFunction > > > > > > > > > >>>>>>> since it > > > > > > > > > >>>>>>> is going away. The longer we do not have this > > > indication, > > > > > the > > > > > > > > more > > > > > > > > > >>>> user > > > > > > > > > >>>>>>> implementations will be based on it and the more > pain > > > > will > > > > > be > > > > > > > > > >>> induced > > > > > > > > > >>>>>> when > > > > > > > > > >>>>>>> we > > > > > > > > > >>>>>>> finally drop it. Users now have an alternative API > > that > > > > > they > > > > > > > > > >>> should > > > > > > > > > >>>> use > > > > > > > > > >>>>>> and > > > > > > > > > >>>>>>> which > > > > > > > > > >>>>>>> is fully functional, from that perspective nothing > > > blocks > > > > > > > marking > > > > > > > > > >>> it > > > > > > > > > >>>>>>> @Deprecated. > > > > > > > > > >>>>>>> As for the remaining work items - there are > primarily > > > > three > > > > > > > > kinds: > > > > > > > > > >>>>>>> > > > > > > > > > >>>>>>> 1. Where Flink internally uses SourceFunction, > > without > > > > > > exposing > > > > > > > > > >>> this > > > > > > > > > >>>>> fact > > > > > > > > > >>>>>>> to the > > > > > > > > > >>>>>>> outside world: > > > > > > > > > >>>>>>> - FLINK-28050 [3] > > > > > > > > > >>>>>>> - FLINK-28229 [4] > > > > > > > > > >>>>>>> - FLINK-28048 [5] > > > > > > > > > >>>>>>> > > > > > > > > > >>>>>>> 2. Very specific edge cases that might not be > covered > > > by > > > > > the > > > > > > > > > >>> Source > > > > > > > > > >>>> API > > > > > > > > > >>>>>> as > > > > > > > > > >>>>>>> is: > > > > > > > > > >>>>>>> - FLINK-28054 [6] > > > > > > > > > >>>>>>> - FLINK-28051 [7] > > > > > > > > > >>>>>>> > > > > > > > > > >>>>>>> 3. Usability improvements - something that was > easily > > > > > doable > > > > > > > with > > > > > > > > > >>>>>>> SourceFunction, > > > > > > > > > >>>>>>> but requires deep knowledge of the new, > > > significantly > > > > > > more > > > > > > > > > >>>> complex, > > > > > > > > > >>>>>>> Source API > > > > > > > > > >>>>>>> to achieve: > > > > > > > > > >>>>>>> - FLINK-28056 [8] > > > > > > > > > >>>>>>> > > > > > > > > > >>>>>>> In my mind, none of those are blockers for > proceeding > > > > with > > > > > > > adding > > > > > > > > > >>> the > > > > > > > > > >>>>>>> @Deprecated > > > > > > > > > >>>>>>> annotation: > > > > > > > > > >>>>>>> (1) is a simple case of encapsulation, internals > > should > > > > not > > > > > > > > > >>> concern > > > > > > > > > >>>> the > > > > > > > > > >>>>>> API > > > > > > > > > >>>>>>> users > > > > > > > > > >>>>>>> (2) is really only relevant for "exotic" use cases. > > > Does > > > > > not > > > > > > > mean > > > > > > > > > >>> we > > > > > > > > > >>>>>> should > > > > > > > > > >>>>>>> not > > > > > > > > > >>>>>>> consider those, but since it is irrelevant for > 99.9% > > of > > > > the > > > > > > > > > >>> users, I > > > > > > > > > >>>> do > > > > > > > > > >>>>>> not > > > > > > > > > >>>>>>> think > > > > > > > > > >>>>>>> we should get stuck here. > > > > > > > > > >>>>>>> (3) is purely a nice to have. Formally speaking, > all > > of > > > > the > > > > > > > tools > > > > > > > > > >>> are > > > > > > > > > >>>>>>> there, it is > > > > > > > > > >>>>>>> just that due to the complexity of the new Source > API > > > > some > > > > > > > > > >>> "simple" > > > > > > > > > >>>>>> things > > > > > > > > > >>>>>>> become > > > > > > > > > >>>>>>> non-trivial and ideally we want to do better here. > > > > > > > > > >>>>>>> > > > > > > > > > >>>>>>> [1] > > > > > > > > > >>> > > > > > https://lists.apache.org/thread/d6cwqw9b3105wcpdkwq7rr4s7x4ywqr9 > > > > > > > > > >>>>>>> [2] > > > > > > > > > >>> > > > > > https://lists.apache.org/thread/kv9rj3w2rmkb8jtss5bqffhw57or7v8v > > > > > > > > > >>>>>>> [3] > > https://issues.apache.org/jira/browse/FLINK-28050 > > > > > > > > > >>>>>>> [4] > > https://issues.apache.org/jira/browse/FLINK-28229 > > > > > > > > > >>>>>>> [5] > > https://issues.apache.org/jira/browse/FLINK-28048 > > > > > > > > > >>>>>>> [6] > > https://issues.apache.org/jira/browse/FLINK-28054 > > > > > > > > > >>>>>>> [7] > > https://issues.apache.org/jira/browse/FLINK-28051 > > > > > > > > > >>>>>>> [8] > > https://issues.apache.org/jira/browse/FLINK-28056 > > > > > > > > > >>>>>>> > > > > > > > > > >>>>>>> On Thu, 29 Jun 2023 at 13:13, Xintong Song < > > > > > > > > tonysong...@gmail.com > > > > > > > > > >>>>>> wrote: > > > > > > > > > >>>>>>>> Thanks for the inputs, Martijn, Jing and Alex. > > > > > > > > > >>>>>>>> > > > > > > > > > >>>>>>>> @Martijn, > > > > > > > > > >>>>>>>> Regarding the Scala supports, I personally don't > > think > > > > "a > > > > > > > fully > > > > > > > > > >>>>> striked > > > > > > > > > >>>>>>>> through experience in the IDE" is something we > want > > to > > > > > > avoid, > > > > > > > > > >>>>>> especially > > > > > > > > > >>>>>>>> given that we are planning to remove the > deprecated > > > APIs > > > > > > soon, > > > > > > > > > >>>> unlike > > > > > > > > > >>>>>>> when > > > > > > > > > >>>>>>>> FLINK-29740 was resolved we didn't really know > when > > > they > > > > > > would > > > > > > > > > >>> be > > > > > > > > > >>>>>>> removed. > > > > > > > > > >>>>>>>> Moreover, the even entry point for DataStream > Scala > > > > > > > > > >>>>>>>> (`StreamExecutionEnvironment`) is not annotated. > > > > > > > > > >>>>>>>> > > > > > > > > > >>>>>>>> > > > > > > > > > >>>>>>>> @Jing and @Alex, > > > > > > > > > >>>>>>>> > > > > > > > > > >>>>>>>> IIUC, you mean SourceFunction can be annotated as > > > > > > > `@Deprecated` > > > > > > > > > >>> in > > > > > > > > > >>>>>> 1.18, > > > > > > > > > >>>>>>>> and there's already a PR doing so. However, after > > the > > > > > > > > > >>> deprecation, > > > > > > > > > >>>>>> there > > > > > > > > > >>>>>>>> are still issues that need to be addressed before > > > > removing > > > > > > it > > > > > > > in > > > > > > > > > >>>> 2.0? > > > > > > > > > >>>>>>> This > > > > > > > > > >>>>>>>> sounds a bit weird. If the API cannot be dropped, > > > which > > > > > > means > > > > > > > > > >>>> without > > > > > > > > > >>>>>>> this > > > > > > > > > >>>>>>>> API some of functions cannot be supported, then > how > > > > could > > > > > it > > > > > > > be > > > > > > > > > >>>>>>> deprecated? > > > > > > > > > >>>>>>>> How would we expect users to migrate away from it? > > > > > > > > > >>>>>>>> > > > > > > > > > >>>>>>>> > > > > > > > > > >>>>>>>> @Jing, > > > > > > > > > >>>>>>>> > > > > > > > > > >>>>>>>> Sounds like it's impractical to deprecate > > SinkFunction > > > > in > > > > > > > 1.18. > > > > > > > > > >>> Any > > > > > > > > > >>>>>>>> expectation / plan on when / how it can be > > deprecated > > > / > > > > > > > removed? > > > > > > > > > >>>>>>>> > > > > > > > > > >>>>>>>> > > > > > > > > > >>>>>>>> Best, > > > > > > > > > >>>>>>>> > > > > > > > > > >>>>>>>> Xintong > > > > > > > > > >>>>>>>> > > > > > > > > > >>>>>>>> > > > > > > > > > >>>>>>>> > > > > > > > > > >>>>>>>> On Thu, Jun 29, 2023 at 6:12 PM Alexander Fedulov > < > > > > > > > > > >>>>>>>> alexander.fedu...@gmail.com> wrote: > > > > > > > > > >>>>>>>> > > > > > > > > > >>>>>>>>> Hi Xintong, > > > > > > > > > >>>>>>>>> > > > > > > > > > >>>>>>>>> Thanks for bringing up this topic. I can provide > > some > > > > > > details > > > > > > > > > >>>>>> regarding > > > > > > > > > >>>>>>>>> the SourceFunction deprecation efforts. Marking > > > > > > > > > >>> SourceFunction as > > > > > > > > > >>>>>>>>> deprecated was not possible until now since we > have > > > > > > stringent > > > > > > > > > >>>>>> compiler > > > > > > > > > >>>>>>>>> checks in flink-examples against using any > > deprecated > > > > > APIs. > > > > > > > We > > > > > > > > > >>>>>> actually > > > > > > > > > >>>>>>>>> merged the migration of all examples to the new > > > > > > FLIP-27-based > > > > > > > > > >>>>>>>>> DataGeneratorSource [1] just two days ago [2]. > Now > > > the > > > > PR > > > > > > > > > >>> marking > > > > > > > > > >>>>>>>>> it @Deprecated is finally unblocked [3] (I would > be > > > > > > grateful > > > > > > > > > >>> if > > > > > > > > > >>>> you > > > > > > > > > >>>>>>> could > > > > > > > > > >>>>>>>>> merge it). > > > > > > > > > >>>>>>>>> > > > > > > > > > >>>>>>>>> With regards to the Flink 2.0 scope, I compiled a > > > list > > > > of > > > > > > > > > >>> items > > > > > > > > > >>>>>>> required > > > > > > > > > >>>>>>>> to > > > > > > > > > >>>>>>>>> be able to drop the SourceFunction API [4] a > while > > > ago > > > > > and > > > > > > as > > > > > > > > > >>> you > > > > > > > > > >>>>> can > > > > > > > > > >>>>>>>> see, > > > > > > > > > >>>>>>>>> there is still quite some work to be done. Some > > items > > > > [5] > > > > > > > > > >>> might > > > > > > > > > >>>>> even > > > > > > > > > >>>>>>>>> require additions to the new Source API. > Overall, I > > > am > > > > > > happy > > > > > > > > > >>> to > > > > > > > > > >>>>> take > > > > > > > > > >>>>>>>>> ownership of completing this work package. > > > > > > > > > >>>>>>>>> > > > > > > > > > >>>>>>>>> Best, > > > > > > > > > >>>>>>>>> Alex > > > > > > > > > >>>>>>>>> > > > > > > > > > >>>>>>>>> > > > > > > > > > >>>>>>>>> [1] https://cwiki.apache.org/confluence/x/9Av1D > > > > > > > > > >>>>>>>>> [2] https://github.com/apache/flink/pull/21774 > > > > > > > > > >>>>>>>>> [3] https://github.com/apache/flink/pull/20049 > > > > > > > > > >>>>>>>>> [4] > > > https://issues.apache.org/jira/browse/FLINK-28045 > > > > > > > > > >>>>>>>>> [5] > > > https://issues.apache.org/jira/browse/FLINK-28054 > > > > > > > > > >>>>>>>>> > > > > > > > > > >>>>>>>>> On Thu, 29 Jun 2023 at 10:45, Martijn Visser < > > > > > > > > > >>>>>> martijnvis...@apache.org > > > > > > > > > >>>>>>>>> wrote: > > > > > > > > > >>>>>>>>> > > > > > > > > > >>>>>>>>>> Hi Xintong, > > > > > > > > > >>>>>>>>>> > > > > > > > > > >>>>>>>>>> With regards to the deprecation of the Scala > APIs, > > > > > during > > > > > > > > > >>> the > > > > > > > > > >>>> PR > > > > > > > > > >>>>>>>>>> review it was requested to not mark all APIs as > > > > > deprecated > > > > > > > > > >>> but > > > > > > > > > >>>>> only > > > > > > > > > >>>>>>>>>> the entry point [1], to avoid a fully striked > > > through > > > > > > > > > >>>> experience > > > > > > > > > >>>>> in > > > > > > > > > >>>>>>>>>> the IDE. I think the same idea was applicable on > > the > > > > > > DataSet > > > > > > > > > >>>>> API. I > > > > > > > > > >>>>>>>>>> think it depends on how formal we want to treat > > > this: > > > > if > > > > > > > > > >>> really > > > > > > > > > >>>>>>>>>> formal, then we should deprecate them in 1.18. I > > > think > > > > > in > > > > > > > > > >>> both > > > > > > > > > >>>>>> cases, > > > > > > > > > >>>>>>>>>> it's quite well known that they are deprecated. > > I'm > > > +0 > > > > > for > > > > > > > > > >>>> either > > > > > > > > > >>>>>>> way, > > > > > > > > > >>>>>>>>>> as long as we're all agreeing that they can be > > > removed > > > > > in > > > > > > > > > >>> 2.0. > > > > > > > > > >>>>>>>>>> With regards to Queryable State and > > > > Source/SinkFunction, > > > > > > +1 > > > > > > > > > >>> to > > > > > > > > > >>>>> mark > > > > > > > > > >>>>>>>>>> these as deprecated. > > > > > > > > > >>>>>>>>>> > > > > > > > > > >>>>>>>>>> Best regards, > > > > > > > > > >>>>>>>>>> > > > > > > > > > >>>>>>>>>> Martijn > > > > > > > > > >>>>>>>>>> > > > > > > > > > >>>>>>>>>> [1] > > > > > > > > > >>>>>>>>>> > > > > > > > > > >>>> > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/apache/flink/pull/21176#pullrequestreview-1159706808 > > > > > > > > > >>>>>>>>>> On Thu, Jun 29, 2023 at 10:23 AM Xintong Song < > > > > > > > > > >>>>>> tonysong...@gmail.com > > > > > > > > > >>>>>>>>>> wrote: > > > > > > > > > >>>>>>>>>>> Hi devs, > > > > > > > > > >>>>>>>>>>> > > > > > > > > > >>>>>>>>>>> Looking at the release 2.0 proposals [1], I > > noticed > > > > > that > > > > > > > > > >>> many > > > > > > > > > >>>>>> APIs > > > > > > > > > >>>>>>>> that > > > > > > > > > >>>>>>>>>> are > > > > > > > > > >>>>>>>>>>> proposed to be removed in 2.0 are not (fully) > > > > > deprecated > > > > > > > > > >>> yet. > > > > > > > > > >>>>> We > > > > > > > > > >>>>>>>> might > > > > > > > > > >>>>>>>>>> want > > > > > > > > > >>>>>>>>>>> to properly mark them as `@Deprecated` in 1.18 > if > > > we > > > > > > agree > > > > > > > > > >>>> they > > > > > > > > > >>>>>>>> should > > > > > > > > > >>>>>>>>> be > > > > > > > > > >>>>>>>>>>> removed in 2.0. Moreover, according to FLIP-321 > > [2] > > > > > (not > > > > > > > > > >>>> voted > > > > > > > > > >>>>>> yet > > > > > > > > > >>>>>>>> but > > > > > > > > > >>>>>>>>>> IMO > > > > > > > > > >>>>>>>>>>> is close to consensus IMO), a migration period > is > > > > > > required > > > > > > > > > >>>>> after > > > > > > > > > >>>>>>> APIs > > > > > > > > > >>>>>>>>> are > > > > > > > > > >>>>>>>>>>> deprecated and before they can be removed. > > > > > > > > > >>>>>>>>>>> > > > > > > > > > >>>>>>>>>>> I might not be familiar with the status of all > > the > > > > APIs > > > > > > > > > >>>> below. > > > > > > > > > >>>>> So > > > > > > > > > >>>>>>> I'd > > > > > > > > > >>>>>>>>>> like > > > > > > > > > >>>>>>>>>>> to bring them up and see if there's any concern > > > > > regarding > > > > > > > > > >>>>>>> deprecating > > > > > > > > > >>>>>>>>>> them > > > > > > > > > >>>>>>>>>>> in 1.18. If there's concern for deprecating > API, > > we > > > > can > > > > > > > > > >>>> start a > > > > > > > > > >>>>>>>>> separate > > > > > > > > > >>>>>>>>>>> discussion thread for it. For those with no > > > > objections, > > > > > > > > > >>> I'd > > > > > > > > > >>>>>> create > > > > > > > > > >>>>>>>> JIRA > > > > > > > > > >>>>>>>>>>> tickets and try to properly deprecate them in > > 1.18. > > > > > > > > > >>>>>>>>>>> > > > > > > > > > >>>>>>>>>>> 1. DataSet API > > > > > > > > > >>>>>>>>>>> It's described as "legacy", "soft deprecated" > in > > > user > > > > > > > > > >>>>>> documentation > > > > > > > > > >>>>>>>>> [3]. > > > > > > > > > >>>>>>>>>>> However, it's not annotated with `@Deprecated` > in > > > > > codes. > > > > > > > > > >>>>>> According > > > > > > > > > >>>>>>> to > > > > > > > > > >>>>>>>>>>> FLIP-131 [4], DataSet API should be deprecated > > when > > > > > > > > > >>>> DataStream > > > > > > > > > >>>>>> API > > > > > > > > > >>>>>>>> and > > > > > > > > > >>>>>>>>>>> Table API / SQL meet certain requirements. > > AFAICS, > > > > all > > > > > > the > > > > > > > > > >>>>>>>> requirements > > > > > > > > > >>>>>>>>>>> mentioned in the FLIP are already fulfilled. We > > > > should > > > > > > > > > >>>> annotate > > > > > > > > > >>>>>> it > > > > > > > > > >>>>>>> as > > > > > > > > > >>>>>>>>>>> `@Deprecated` now. > > > > > > > > > >>>>>>>>>>> > > > > > > > > > >>>>>>>>>>> 2. SourceFunction / SinkFunction > > > > > > > > > >>>>>>>>>>> They are described as deprecated in the > > roadmap[5], > > > > > and I > > > > > > > > > >>>> don't > > > > > > > > > >>>>>>> find > > > > > > > > > >>>>>>>>>>> anything regarding them in user documentation. > > But > > > > they > > > > > > > > > >>> are > > > > > > > > > >>>>> also > > > > > > > > > >>>>>>> not > > > > > > > > > >>>>>>>>>>> annotated with `@Deprecated` in codes. TBH, I'm > > not > > > > > aware > > > > > > > > > >>> of > > > > > > > > > >>>>> any > > > > > > > > > >>>>>>>> formal > > > > > > > > > >>>>>>>>>>> decision to deprecate these. AFAICS, the > > > replacement > > > > > for > > > > > > > > > >>>>>>>> SourceFunction > > > > > > > > > >>>>>>>>>>> (Source) has already been promoted to > `@Public`, > > > > while > > > > > > the > > > > > > > > > >>>>>>>> replacement > > > > > > > > > >>>>>>>>>> for > > > > > > > > > >>>>>>>>>>> SinkFunction (SinkV2) is still > > `@PublicEvolving`. I > > > > > found > > > > > > > > > >>> a > > > > > > > > > >>>>>>>>> discussion[6] > > > > > > > > > >>>>>>>>>>> regarding promoting SinkV2 to `@Public`, but > it's > > > > > unclear > > > > > > > > > >>> to > > > > > > > > > >>>> me > > > > > > > > > >>>>>>> what > > > > > > > > > >>>>>>>>> the > > > > > > > > > >>>>>>>>>>> conclusion is. > > > > > > > > > >>>>>>>>>>> > > > > > > > > > >>>>>>>>>>> 3. Queryable State > > > > > > > > > >>>>>>>>>>> It's described as approaching end-of-life in > the > > > > > roadmap > > > > > > > > > >>> [5], > > > > > > > > > >>>>> but > > > > > > > > > >>>>>>> is > > > > > > > > > >>>>>>>>>>> neither deprecated in codes nor in user > > > documentation > > > > > > > > > >>> [7]. I > > > > > > > > > >>>>> also > > > > > > > > > >>>>>>>>> found a > > > > > > > > > >>>>>>>>>>> discussion [8] about rescuing it from > > deprecation, > > > > and > > > > > it > > > > > > > > > >>>> seems > > > > > > > > > >>>>>> to > > > > > > > > > >>>>>>> me > > > > > > > > > >>>>>>>>>> there > > > > > > > > > >>>>>>>>>>> are more negative opinions than positive ones. > > > > > > > > > >>>>>>>>>>> > > > > > > > > > >>>>>>>>>>> 4. All Scala APIs > > > > > > > > > >>>>>>>>>>> I think we agreed to drop Scala API support in > > > > FLIP-265 > > > > > > > > > >>> [9], > > > > > > > > > >>>>> and > > > > > > > > > >>>>>>> have > > > > > > > > > >>>>>>>>>> tried > > > > > > > > > >>>>>>>>>>> to deprecate them in FLINK-29740 [10]. Also, > both > > > > user > > > > > > > > > >>>>>>> documentation > > > > > > > > > >>>>>>>>> and > > > > > > > > > >>>>>>>>>>> roadmap[5] shows that scala API supports are > > > > > deprecated. > > > > > > > > > >>>>> However, > > > > > > > > > >>>>>>>>> AFAICS, > > > > > > > > > >>>>>>>>>>> none of the APIs in `flink-streaming-scala` are > > > > > annotated > > > > > > > > > >>>> with > > > > > > > > > >>>>>>>>>>> `@Deprecated`, and only `ExecutionEnvironment` > > and > > > > > > > > > >>> `package` > > > > > > > > > >>>>> are > > > > > > > > > >>>>>>>> marked > > > > > > > > > >>>>>>>>>>> `@Deprecated` in `flink-scala`. > > > > > > > > > >>>>>>>>>>> > > > > > > > > > >>>>>>>>>>> Looking forward to your feedback. > > > > > > > > > >>>>>>>>>>> > > > > > > > > > >>>>>>>>>>> Best, > > > > > > > > > >>>>>>>>>>> > > > > > > > > > >>>>>>>>>>> Xintong > > > > > > > > > >>>>>>>>>>> > > > > > > > > > >>>>>>>>>>> > > > > > > > > > >>>>>>>>>>> [1] > > > > > > > > > >>>>>> > > > > > https://cwiki.apache.org/confluence/display/FLINK/2.0+Release > > > > > > > > > >>>>>>>>>>> [2] > > > > > > > > > >>>>>>> > > > > > > > > https://lists.apache.org/thread/vmhzv8fcw2b33pqxp43486owrxbkd5x9 > > > > > > > > > >>>>>>>>>>> [3] > > > > > > > > > >>>>>>>>>>> > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://nightlies.apache.org/flink/flink-docs-master/docs/dev/dataset/overview/ > > > > > > > > > >>>>>>>>>>> [4] > > > > > > > > > >>>>>>>>>>> > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=158866741 > > > > > > > > > >>>>>>>>>>> [5] https://flink.apache.org/roadmap/ > > > > > > > > > >>>>>>>>>>> > > > > > > > > > >>>>>>>>>>> [6] > > > > > > > > > >>>>>>> > > > > > > > > https://lists.apache.org/thread/q62nj89rrz0t5xtggy5n65on95f2rmmx > > > > > > > > > >>>>>>>>>>> [7] > > > > > > > > > >>>>>>>>>>> > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/fault-tolerance/queryable_state/ > > > > > > > > > >>>>>>>>>>> [8] > > > > > > > > > >>>>>>> > > > > > > > > https://lists.apache.org/thread/9hmwcjb3q5c24pk3qshjvybfqk62v17m > > > > > > > > > >>>>>>>>>>> [9] > > > > > > > > > >>>>>>>>>>> > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-265+Deprecate+and+remove+Scala+API+support > > > > > > > > > >>>>>>>>>>> [10] > > > > https://issues.apache.org/jira/browse/FLINK-29740 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
