Hi, Jing, I share the same opinion with Xintong. The base class `TimestampExtractor` for these two classes was deprecated in release-1.12[1][2], so deprecating these two subclasses is relatively straightforward.
[1] https://issues.apache.org/jira/browse/FLINK-19453 [2] https://cwiki.apache.org/confluence/display/FLINK/FLIP-95%3A+New+TableSource+and+TableSink+interfaces Best, Jane On Fri, Jul 21, 2023 at 10:19 AM Xintong Song <tonysong...@gmail.com> wrote: > I personally don't think it's necessary. IIUC, we have already decided to > deprecate TableSource / TableSink, and not marking StreamRecordTimestamp > and ExistingField deprecated is just an oversight in carrying out that > decision. > > > Best, > > Xintong > > > > On Fri, Jul 21, 2023 at 7:48 AM Jing Ge <j...@ververica.com.invalid> > wrote: > > > Hi Jane, Hi Xintong, > > > > The specific discussion of deprecating `StreamRecordTimestamp` & > > `ExistingField` was kind of hiding in the discussion of the umbrella > topic > > that people might miss out. Does it make sense to start an > > individual Discussion thread for it like we deprecate any other APIs? > > > > For all APIs that should be marked as @PublicEvolving, dedicated > graduation > > threads are recommended. > > > > Best regards, > > Jing > > > > > > On Wed, Jul 19, 2023 at 10:09 AM Xintong Song <tonysong...@gmail.com> > > wrote: > > > > > Thank you, Jane. > > > > > > What you said (preparing a FLIP for ManagedTable related classes, not > > > deprecating SourceFunctionProvider, and starting a dedicated discussion > > for > > > the missing annotations) sounds good to me. > > > > > > In addition, if there's no objections from the community on marking > > > `StreamRecordTimestamp` & `ExistingField` as deprecated, you may > consider > > > creating a Jira ticket for it as a sub-task of FLINK-32557. > > > > > > Best, > > > > > > Xintong > > > > > > > > > > > > On Wed, Jul 19, 2023 at 3:48 PM Jane Chan <qingyue....@gmail.com> > wrote: > > > > > > > Hi Xintong, > > > > > > > > > IIUC, you are suggesting to mark the classes with a red background > as > > > > `@Deprecated` in 1.18? > > > > > > > > Exactly. They are not marked at the code level but suppose to be > > > > deprecated. > > > > > > > > > - For the ManagedTable related classes, is there any FLIP > explicitly > > > that > > > > decides to deprecate them? If not, I think it would be nice to have > > one, > > > to > > > > formally decide the deprecation with a vote. I'd expect such a FLIP > can > > > be > > > > quite lightweight. > > > > > > > > Currently, there's no formal FLIP yet. I'd like to prepare one to > > > initiate > > > > the deprecation process. > > > > > > > > > - For `SourceFunctionProvider`, please beware there are recently > some > > > > objections against deprecating `SourceFunction`, and the code changes > > > > marking it as `@Deprecated` might be reverted. See [1][2] for more > > > details. > > > > So my question is, if `SourceFunction` will not be deprecated until > all > > > > sub-tasks in FLINK-28045 are resolved, do you still think we should > > > > deprecate `SourceFunctionProvider` now? > > > > > > > > Thanks for the reminder. If `SourceFunction` is no longer being > > > deprecated, > > > > then `SourceFunctionProvider` also needs to be retained. I have > updated > > > the > > > > sheet and removed `SourceFunctionProvider`. > > > > > > > > > I'm in general +1 to add the missing API annotations. However, I > > don't > > > > have the expertise to comment on the classes and suggestd API levels > > > being > > > > listed. > > > > > > > > I've updated the sheet and added comments on all the APIs that are > > > > suggested to be marked as PublicEvolving, explaining the reasons. The > > > rest > > > > APIs are either util classes or implementations and hence are > suggested > > > to > > > > be Internal. I can start a discussion about the suggested API level > to > > > find > > > > developers who can help review them. > > > > > > > > Best, > > > > Jane > > > > > > > > On Wed, Jul 19, 2023 at 12:22 PM Xintong Song <tonysong...@gmail.com > > > > > > wrote: > > > > > > > > > Thanks for the beautiful sheets, Jane. > > > > > > > > > > 1. This sheet < > > > > > > > > > > > > > > > > > > > > > https://docs.google.com/spreadsheets/d/1dZBNHLuAHYJt3pFU8ZtfUzrYyf2ZFQ6wybDXGS1bHno/edit?usp=sharing > > > > > > > > > > > > summarizes the user-facing classes and methods that need to be > > > > deprecated > > > > > > under the flink-table module, some of which are marked with a red > > > > > > background and belong to the APIs that need to be depreciated but > > are > > > > not > > > > > > explicitly marked in the code. This mainly includes legacy table > > > > > > source/sink, legacy table schema, legacy SQL function, and some > > > > internal > > > > > > APIs designed for Paimon but are now obsolete. > > > > > > > > > > > IIUC, you are suggesting to mark the classes with a red background > as > > > > > `@Deprecated` in 1.18? > > > > > > > > > > - +1 for deprecating `StreamRecordTimestamp` & `ExistingField` > in > > > > 1.18. > > > > > Based on your description, it seems these were not marked by > > > mistake. > > > > > Let's > > > > > fix them. > > > > > - For the ManagedTable related classes, is there any FLIP > > explicitly > > > > > that decides to deprecate them? If not, I think it would be nice > > to > > > > have > > > > > one, to formally decide the deprecation with a vote. I'd expect > > > such a > > > > > FLIP > > > > > can be quite lightweight. > > > > > - For `SourceFunctionProvider`, please beware there are recently > > > some > > > > > objections against deprecating `SourceFunction`, and the code > > > changes > > > > > marking it as `@Deprecated` might be reverted. See [1][2] for > more > > > > > details. > > > > > So my question is, if `SourceFunction` will not be deprecated > > until > > > > all > > > > > sub-tasks in FLINK-28045 are resolved, do you still think we > > should > > > > > deprecate `SourceFunctionProvider` now? > > > > > > > > > > 2. In addition, during the process of organizing, it was found that > > > some > > > > > > APIs under the flink-table-api-java and flink-table-common > modules > > do > > > > not > > > > > > have an explicit API annotation (you can find detailed > information > > in > > > > > this > > > > > > sheet < > > > > > > > > > > > > > > > > > > > > > https://docs.google.com/spreadsheets/d/1e8M0tUtKkZXEd8rCZtZ0C6Ty9QkNaPySsrCgz0vEID4/edit?usp=sharing > > > > > >). > > > > > > I suggest explicitly marking the level for these APIs. > > > > > > > > > > I'm in general +1 to add the missing API annotations. However, I > > don't > > > > have > > > > > the expertise to comment on the classes and suggestd API levels > being > > > > > listed. > > > > > > > > > > 3. As there are still some internal and test code dependencies on > > these > > > > > > APIs, can we first gradually migrate these dependencies to > > > alternative > > > > > > APIs to make the deprecation process relatively easy? > > > > > > > > > > > That makes sense to me. I think the purpose of trying to mark the > > APIs > > > as > > > > > deprecated in 1.18 is to send users the signal early that these > APIs > > > will > > > > > be removed. As for the internal and test code dependencies, I don't > > see > > > > any > > > > > problem in gradually migrating them. > > > > > > > > > > Best, > > > > > > > > > > Xintong > > > > > > > > > > > > > > > [1] > https://lists.apache.org/thread/734zhkvs59w2o4d1rsnozr1bfqlr6rgm > > > > > > > > > > [2] https://issues.apache.org/jira/browse/FLINK-28046 > > > > > > > > > > > > > > > > > > > > On Wed, Jul 19, 2023 at 11:41 AM Jane Chan <qingyue....@gmail.com> > > > > wrote: > > > > > > > > > > > Hi Xintong, > > > > > > > > > > > > Thanks for driving this topic. Regarding the Table API > > deprecation, I > > > > can > > > > > > provide some details to help with the process. > > > > > > > > > > > > 1. This sheet > > > > > > < > > > > > > > > > > > > > > > > > > > > > https://docs.google.com/spreadsheets/d/1dZBNHLuAHYJt3pFU8ZtfUzrYyf2ZFQ6wybDXGS1bHno/edit?usp=sharing > > > > > > > > > > > > > summarizes > > > > > > the user-facing classes and methods that need to be deprecated > > > under > > > > > the > > > > > > flink-table module, some of which are marked with a red > > background > > > > and > > > > > > belong to the APIs that need to be depreciated but are not > > > > explicitly > > > > > > marked in the code. This mainly includes legacy table > > source/sink, > > > > > > legacy > > > > > > table schema, legacy SQL function, and some internal APIs > > designed > > > > for > > > > > > Paimon but are now obsolete. > > > > > > 2. In addition, during the process of organizing, it was found > > > that > > > > > some > > > > > > APIs under the flink-table-api-java and flink-table-common > > modules > > > > do > > > > > > not > > > > > > have an explicit API annotation (you can find detailed > > information > > > > in > > > > > > this > > > > > > sheet > > > > > > < > > > > > > > > > > > > > > > > > > > > > https://docs.google.com/spreadsheets/d/1e8M0tUtKkZXEd8rCZtZ0C6Ty9QkNaPySsrCgz0vEID4/edit?usp=sharing > > > > > > >). > > > > > > I suggest explicitly marking the level for these APIs. > > > > > > 3. As there are still some internal and test code dependencies > > on > > > > > these > > > > > > APIs, can we first gradually migrate these dependencies to > > > > > alternative > > > > > > APIs to make the deprecation process relatively easy? > > > > > > > > > > > > I hope this helps. > > > > > > > > > > > > Best, > > > > > > Jane > > > > > > > > > > > > On Tue, Jul 11, 2023 at 12:08 PM Xintong Song < > > tonysong...@gmail.com > > > > > > > > > > wrote: > > > > > > > > > > > > > Thanks for bringing this up, Matthias. I've replied to you in > the > > > > other > > > > > > > thread[1]. > > > > > > > > > > > > > > Best, > > > > > > > > > > > > > > Xintong > > > > > > > > > > > > > > > > > > > > > [1] > > > https://lists.apache.org/thread/l3dkdypyrovd3txzodn07lgdwtwvhgk4 > > > > > > > > > > > > > > On Mon, Jul 10, 2023 at 8:37 PM Matthias Pohl > > > > > > > <matthias.p...@aiven.io.invalid> wrote: > > > > > > > > > > > > > > > @Xintong did you come across FLINK-3957 [1] when looking for > > > > > deprecated > > > > > > > > issues? There are a few subtasks that would require > > deprecations > > > as > > > > > > well > > > > > > > as > > > > > > > > far as I can see. For some I'm wondering whether we should > add > > > them > > > > > to > > > > > > > the > > > > > > > > release 2.0 feature list [2] in some way and (as a > consequence) > > > > > missed > > > > > > > them > > > > > > > > in FLINK-32557 [3], e.g.: > > > > > > > > - FLINK-4675 [4] > > > > > > > > - FLINK-14068 [5] (listed in the release 2.0 feature list [2] > > but > > > > not > > > > > > > > listed in FLINK-32557 [3]) > > > > > > > > - FLINK-5126 [6] > > > > > > > > - FLINK-13926 [7] > > > > > > > > > > > > > > > > For some other subtasks, I'm not 100% sure whether they still > > > > apply. > > > > > > > Others > > > > > > > > might resolve by removing the DataSet or Scala API. But for > > the 4 > > > > > > > mentioned > > > > > > > > above we might need to deprecate API in 1.18. > > > > > > > > > > > > > > > > Matthias > > > > > > > > > > > > > > > > [1] https://issues.apache.org/jira/browse/FLINK-3957 > > > > > > > > [2] > > > https://cwiki.apache.org/confluence/display/FLINK/2.0+Release > > > > > > > > [3] https://issues.apache.org/jira/browse/FLINK-32557 > > > > > > > > > > > > > > > > [4] https://issues.apache.org/jira/browse/FLINK-4675 > > > > > > > > [5] https://issues.apache.org/jira/browse/FLINK-14068 > > > > > > > > [6] https://issues.apache.org/jira/browse/FLINK-5126 > > > > > > > > [7] https://issues.apache.org/jira/browse/FLINK-13926 > > > > > > > > > > > > > > > > On Fri, Jul 7, 2023 at 12:59 PM Jing Ge > > > <j...@ververica.com.invalid > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > Hi Alex, > > > > > > > > > > > > > > > > > > I would follow FLIP-197 and try to release them asap > > depending > > > on > > > > > dev > > > > > > > > > resources and how difficult those issues are. The fastest > > > > timeline > > > > > is > > > > > > > the > > > > > > > > > period defined in FLIP-197 in ideal conditions. > > > > > > > > > > > > > > > > > > Best regards, > > > > > > > > > Jing > > > > > > > > > > > > > > > > > > On Fri, Jul 7, 2023 at 12:20 PM Alexander Fedulov < > > > > > > > > > alexander.fedu...@gmail.com> wrote: > > > > > > > > > > > > > > > > > > > @Xintong > > > > > > > > > > > - IIUC, the testing scenario you described is like > > blocking > > > > the > > > > > > > > source > > > > > > > > > > for > > > > > > > > > > > proceeding (emit data, finish, etc.) until a checkpoint > > is > > > > > > > finished. > > > > > > > > > > > > > > > > > > > > It is more tricky than that - we need to prevent the Sink > > > from > > > > > > > > receiving > > > > > > > > > a > > > > > > > > > > checkpoint barrier until the Source is done emitting a > > given > > > > set > > > > > of > > > > > > > > > > records. In > > > > > > > > > > the current tests, which are also used for V2 Sinks, > > > > > SourceFunction > > > > > > > > > > controls > > > > > > > > > > when the Sink is "allowed" to commit by holding the > > > checkpoint > > > > > lock > > > > > > > > while > > > > > > > > > > producing the records. The lock is not available in the > new > > > > > Source > > > > > > by > > > > > > > > > > design > > > > > > > > > > and we need a solution that provides the same > functionality > > > > > > (without > > > > > > > > > > modifying > > > > > > > > > > the Sinks). I am currently checking if a workaround is at > > all > > > > > > > possible > > > > > > > > > > without > > > > > > > > > > adjusting anything in the Source interface. > > > > > > > > > > > > > > > > > > > > > I may not have understood all the details, but based on > > > what > > > > > you > > > > > > > > > > described > > > > > > > > > > > I'd hesitate to block the deprecation / removal of > > > > > SourceFunction > > > > > > > on > > > > > > > > > > this. > > > > > > > > > > > > > > > > > > > > I don't think we should, just wanted to highlight that > > there > > > > are > > > > > > some > > > > > > > > > > unknowns > > > > > > > > > > with respect to estimating the amount of work required. > > > > > > > > > > > > > > > > > > > > @Jing > > > > > > > > > > I want to understand in which release would you target > > > > graduation > > > > > > of > > > > > > > > the > > > > > > > > > > mentioned connectors to @Public/@PublicEvolving - > basically > > > the > > > > > > > > > anticipated > > > > > > > > > > timeline of the steps in both options with respect to > > > releases. > > > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > > Alex > > > > > > > > > > > > > > > > > > > > On Fri, 7 Jul 2023 at 10:53, Xintong Song < > > > > tonysong...@gmail.com > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > Thanks all for the discussion. I've created FLINK-32557 > > for > > > > > this. > > > > > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > > > > > > > > > > > > > > Xintong > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Jul 6, 2023 at 1:00 AM Jing Ge > > > > > > <j...@ververica.com.invalid > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > Hi Alex, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 3. remove SinkFunction. > > > > > > > > > > > > > Which steps do you imply for the 1.18 release and > for > > > the > > > > > 2.0 > > > > > > > > > > release? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > for 2.0 release. 1.18 will be released soon. > > > > > > > > > > > > > > > > > > > > > > > > Best regards, > > > > > > > > > > > > Jing > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Jul 5, 2023 at 1:08 PM Alexander Fedulov < > > > > > > > > > > > > alexander.fedu...@gmail.com> wrote: > > > > > > > > > > > > > > > > > > > > > > > > > @Jing > > > > > > > > > > > > > Just to clarify, when you say: > > > > > > > > > > > > > > > > > > > > > > > > > > 3. remove SinkFunction. > > > > > > > > > > > > > Which steps do you imply for the 1.18 release and > for > > > the > > > > > 2.0 > > > > > > > > > > release? > > > > > > > > > > > > > > > > > > > > > > > > @Xintong > > > > > > > > > > > > > A side note - with the new Source API we lose the > > > ability > > > > > to > > > > > > > > > control > > > > > > > > > > > > > checkpointing from the source since there is no > lock > > > > > anymore. > > > > > > > > This > > > > > > > > > > > > > functionality > > > > > > > > > > > > > is currently used in a variety of tests for the > > Sinks - > > > > the > > > > > > > tests > > > > > > > > > > that > > > > > > > > > > > > rely > > > > > > > > > > > > > on tight > > > > > > > > > > > > > synchronization between specific elements passed > from > > > the > > > > > > > source > > > > > > > > > to > > > > > > > > > > > the > > > > > > > > > > > > > sink before > > > > > > > > > > > > > allowing a checkpoint to complete (see > > FiniteTestSource > > > > > [1]). > > > > > > > > Since > > > > > > > > > > > > FLIP-27 > > > > > > > > > > > > > Sources rely > > > > > > > > > > > > > on decoupling via the mailbox, without exposing the > > > lock, > > > > > it > > > > > > is > > > > > > > > not > > > > > > > > > > > > > immediately clear > > > > > > > > > > > > > if it is possible to achieve the same functionality > > > > without > > > > > > > major > > > > > > > > > > > > > extensions in the > > > > > > > > > > > > > runtime for such testing purposes. My hope > initially > > > was > > > > > that > > > > > > > > only > > > > > > > > > > the > > > > > > > > > > > > > legacy Sinks > > > > > > > > > > > > > relied on this - this would have made it possible > to > > > drop > > > > > > > > > > > > > SourceFunction+SinkFunction > > > > > > > > > > > > > together, but, in fact, it also already became part > > of > > > > the > > > > > > new > > > > > > > > > SinkV2 > > > > > > > > > > > > > testing IT suits > > > > > > > > > > > > > [2]. Moreover, I know of at least one major > connector > > > > that > > > > > > also > > > > > > > > > > relies > > > > > > > > > > > on > > > > > > > > > > > > > it for > > > > > > > > > > > > > verifying committed sink metadata for a specific > set > > of > > > > > > records > > > > > > > > > > > (Iceberg) > > > > > > > > > > > > > [3]. In my > > > > > > > > > > > > > estimation this currently presents a major blocker > > for > > > > the > > > > > > > > > > > SourceFunction > > > > > > > > > > > > > removal. > > > > > > > > > > > > > > > > > > > > > > > > > > [1] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/apache/flink/blob/master/flink-test-utils-parent/flink-test-utils/src/main/java/org/apache/flink/streaming/util/FiniteTestSource.java > > > > > > > > > > > > > [2] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-files/src/test/java/org/apache/flink/connector/file/sink/StreamingExecutionFileSinkITCase.java#L132 > > > > > > > > > > > > > [3] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/apache/iceberg/blob/master/flink/v1.17/flink/src/test/java/org/apache/iceberg/flink/source/BoundedTestSource.java#L75C1-L85C2 > > > > > > > > > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > > > > > Alex > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, 5 Jul 2023 at 10:47, Chesnay Schepler < > > > > > > > > ches...@apache.org> > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > There's a whole bunch of metric APIs that would > > need > > > to > > > > > be > > > > > > > > > > > deprecated. > > > > > > > > > > > > > > That is of course if the metric FLIPs are being > > > > accepted. > > > > > > > > > > > > > > > > > > > > > > > > > > > > Which makes me wonder if we aren't doing things > the > > > > wrong > > > > > > way > > > > > > > > > > around; > > > > > > > > > > > > > > shouldn't the decision to deprecate an API be > part > > of > > > > the > > > > > > > FLIP > > > > > > > > > > > > > discussion? > > > > > > > > > > > > > > > > > > > > > > > > > > > > On 05/07/2023 07:39, Xintong Song wrote: > > > > > > > > > > > > > > > Thanks all for the discussion. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > It seems to me there's a consensus on marking > the > > > > > > following > > > > > > > > as > > > > > > > > > > > > > deprecated > > > > > > > > > > > > > > > in 1.18: > > > > > > > > > > > > > > > - DataSet API > > > > > > > > > > > > > > > - SourceFunction > > > > > > > > > > > > > > > - Queryable State > > > > > > > > > > > > > > > - All Scala APIs > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > More time is needed for deprecating > SinkFunction. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I'll leave this discussion open for a few more > > > days. > > > > > And > > > > > > if > > > > > > > > > > there's > > > > > > > > > > > > no > > > > > > > > > > > > > > > objections, I'll create JIRA tickets > accordingly. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Xintong > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Jul 5, 2023 at 1:34 PM Xintong Song < > > > > > > > > > > tonysong...@gmail.com > > > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> Thanks for the input, Jing. I'd also be +1 for > > > > option > > > > > 1. > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> Best, > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> Xintong > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> On Mon, Jul 3, 2023 at 7:20 PM Jing Ge > > > > > > > > > > <j...@ververica.com.invalid > > > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >>> Hi Xingtong, > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > >>> Option 1, secure plan would be: > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > >>> 1. graduate kafka, File, JDBC connectors to > > > @Public > > > > > > > > > > > > > > >>> 2. graduate SinkV2 to @Public > > > > > > > > > > > > > > >>> 3. remove SinkFunction. > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > >>> Option 2, risky plan but at a fast pace: > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > >>> 1. graduate SinkV2 to @Public and expecting > > more > > > > > > > > maintenance > > > > > > > > > > > effort > > > > > > > > > > > > > > since > > > > > > > > > > > > > > >>> there are many known and unsolved issues. > > > > > > > > > > > > > > >>> 2. remove SinkFunction. > > > > > > > > > > > > > > >>> 3. It depends on the connectors' contributors > > > > whether > > > > > > > > > > connectors > > > > > > > > > > > > can > > > > > > > > > > > > > > >>> upgrade to Flink 2.0, since we moved forward > > with > > > > > > SinkV2 > > > > > > > > API > > > > > > > > > > > > without > > > > > > > > > > > > > > >>> taking > > > > > > > > > > > > > > >>> care of implementations in external > connectors. > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > >>> I am ok with both of them and personally > prefer > > > > > option > > > > > > 1. > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > >>> Best regards, > > > > > > > > > > > > > > >>> Jing > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > >>> On Fri, Jun 30, 2023 at 3:41 AM Xintong Song > < > > > > > > > > > > > > tonysong...@gmail.com> > > > > > > > > > > > > > > >>> wrote: > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > >>>> I see. Thanks for the explanation. I may > have > > > not > > > > > > looked > > > > > > > > > into > > > > > > > > > > > this > > > > > > > > > > > > > > >>> deeply > > > > > > > > > > > > > > >>>> enough, and would trust the decision from > you > > > and > > > > > the > > > > > > > > > > community > > > > > > > > > > > > > > members > > > > > > > > > > > > > > >>> who > > > > > > > > > > > > > > >>>> participated in the discussion & vote. > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > > > >>>> Best, > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > > > >>>> Xintong > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > > > >>>> On Thu, Jun 29, 2023 at 10:28 PM Alexander > > > > Fedulov < > > > > > > > > > > > > > > >>>> alexander.fedu...@gmail.com> wrote: > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > > > >>>>>> However, I'm not sure about 2. > > > > > > > > > > > > > > >>>>> I am not aware of a bylaw that states the > > > > specific > > > > > > > > > > requirements > > > > > > > > > > > > in > > > > > > > > > > > > > > >>> order > > > > > > > > > > > > > > >>>> to > > > > > > > > > > > > > > >>>>> mark something as @Deprecated. My > > understanding > > > > > from > > > > > > > the > > > > > > > > > > > > discussion > > > > > > > > > > > > > > >>> and > > > > > > > > > > > > > > >>>> the > > > > > > > > > > > > > > >>>>> vote was that the community recognizes the > > > > > necessity > > > > > > to > > > > > > > > > make > > > > > > > > > > it > > > > > > > > > > > > > > >>> explicit > > > > > > > > > > > > > > >>>>> that > > > > > > > > > > > > > > >>>>> the usage of the SourceFunction API is > > > > discouraged. > > > > > > > This > > > > > > > > > can > > > > > > > > > > > > > actually > > > > > > > > > > > > > > >>>>> stimulate > > > > > > > > > > > > > > >>>>> authors of connectors that rely on this > very > > > > > specific > > > > > > > and > > > > > > > > > > > > > > non-baseline > > > > > > > > > > > > > > >>>>> functionality to contribute extensions to > the > > > new > > > > > > > Source > > > > > > > > > API > > > > > > > > > > > > > > >>> themselves > > > > > > > > > > > > > > >>>> in > > > > > > > > > > > > > > >>>>> order to > > > > > > > > > > > > > > >>>>> close the gap. ExternallyInducedSource, for > > > > > instance, > > > > > > > was > > > > > > > > > > > driven > > > > > > > > > > > > by > > > > > > > > > > > > > > >>>> Pravega > > > > > > > > > > > > > > >>>>> to > > > > > > > > > > > > > > >>>>> begin with, since it was only needed for > > their > > > > > > purposes > > > > > > > > > [1]. > > > > > > > > > > We > > > > > > > > > > > > are > > > > > > > > > > > > > > >>> not > > > > > > > > > > > > > > >>>>> removing > > > > > > > > > > > > > > >>>>> anything - until 2.0 everything will > continue > > > to > > > > > work > > > > > > > and > > > > > > > > > we > > > > > > > > > > > can > > > > > > > > > > > > > work > > > > > > > > > > > > > > >>> on > > > > > > > > > > > > > > >>>>> resolving the limitations until then, I > > > > personally > > > > > > > don't > > > > > > > > > see > > > > > > > > > > a > > > > > > > > > > > > big > > > > > > > > > > > > > > >>> issue > > > > > > > > > > > > > > >>>>> here. > > > > > > > > > > > > > > >>>>> > > > > > > > > > > > > > > >>>>>> Do you think it is feasible to resolve > them > > by > > > > the > > > > > > > > feature > > > > > > > > > > > > freeze > > > > > > > > > > > > > > >>> date > > > > > > > > > > > > > > >>>> of > > > > > > > > > > > > > > >>>>> 1.18? > > > > > > > > > > > > > > >>>>> No, these are rather complex additions that > > > would > > > > > > > > probably > > > > > > > > > > > > require > > > > > > > > > > > > > > >>>> FLIP(s). > > > > > > > > > > > > > > >>>>> [1] > > > > > > > > > > > > > > >>>>> > > > > > > > > > > > > > > >>>>> > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://flink.apache.org/2022/01/20/pravega-flink-connector-101/#checkpoint-integration > > > > > > > > > > > > > > >>>>> On Thu, 29 Jun 2023 at 14:25, Xintong Song > < > > > > > > > > > > > > tonysong...@gmail.com> > > > > > > > > > > > > > > >>>> wrote: > > > > > > > > > > > > > > >>>>>> Thanks for the explanation, Alex. > > > > > > > > > > > > > > >>>>>> > > > > > > > > > > > > > > >>>>>> Not blocking the deprecation on 1 & 3 > makes > > > > sense > > > > > to > > > > > > > me. > > > > > > > > > > > > However, > > > > > > > > > > > > > > >>> I'm > > > > > > > > > > > > > > >>>> not > > > > > > > > > > > > > > >>>>>> sure about 2. > > > > > > > > > > > > > > >>>>>> > > > > > > > > > > > > > > >>>>>> It sounds to me that, without FLINK-28051 > & > > > > > > > FLINK-28054, > > > > > > > > > > some > > > > > > > > > > > of > > > > > > > > > > > > > the > > > > > > > > > > > > > > >>>>>> connectors cannot migrate to the new > Source > > > API, > > > > > or > > > > > > at > > > > > > > > > least > > > > > > > > > > > > > further > > > > > > > > > > > > > > >>>>>> investigation is needed to understand the > > > > > situation. > > > > > > > If > > > > > > > > > this > > > > > > > > > > > is > > > > > > > > > > > > > the > > > > > > > > > > > > > > >>>> case, > > > > > > > > > > > > > > >>>>>> we probably should not deprecate the API > > until > > > > > these > > > > > > > > > issues > > > > > > > > > > > are > > > > > > > > > > > > > > >>>> resolved. > > > > > > > > > > > > > > >>>>>> Do you think it is feasible to resolve > them > > by > > > > the > > > > > > > > feature > > > > > > > > > > > > freeze > > > > > > > > > > > > > > >>> date > > > > > > > > > > > > > > >>>> of > > > > > > > > > > > > > > >>>>>> 1.18? > > > > > > > > > > > > > > >>>>>> > > > > > > > > > > > > > > >>>>>> Best, > > > > > > > > > > > > > > >>>>>> > > > > > > > > > > > > > > >>>>>> Xintong > > > > > > > > > > > > > > >>>>>> > > > > > > > > > > > > > > >>>>>> > > > > > > > > > > > > > > >>>>>> > > > > > > > > > > > > > > >>>>>> On Thu, Jun 29, 2023 at 8:02 PM Alexander > > > > Fedulov > > > > > < > > > > > > > > > > > > > > >>>>>> alexander.fedu...@gmail.com> wrote: > > > > > > > > > > > > > > >>>>>> > > > > > > > > > > > > > > >>>>>>> @Xintong > > > > > > > > > > > > > > >>>>>>> The original discussion [1] and vote [2] > > > > > converged > > > > > > on > > > > > > > > the > > > > > > > > > > > idea > > > > > > > > > > > > > > >>> that > > > > > > > > > > > > > > >>>> it > > > > > > > > > > > > > > >>>>> is > > > > > > > > > > > > > > >>>>>>> better > > > > > > > > > > > > > > >>>>>>> to make it clear to the users that they > > > should > > > > > stop > > > > > > > > using > > > > > > > > > > > > > > >>>>> SourceFunction > > > > > > > > > > > > > > >>>>>>> since it > > > > > > > > > > > > > > >>>>>>> is going away. The longer we do not have > > this > > > > > > > > indication, > > > > > > > > > > the > > > > > > > > > > > > > more > > > > > > > > > > > > > > >>>> user > > > > > > > > > > > > > > >>>>>>> implementations will be based on it and > the > > > > more > > > > > > pain > > > > > > > > > will > > > > > > > > > > be > > > > > > > > > > > > > > >>> induced > > > > > > > > > > > > > > >>>>>> when > > > > > > > > > > > > > > >>>>>>> we > > > > > > > > > > > > > > >>>>>>> finally drop it. Users now have an > > > alternative > > > > > API > > > > > > > that > > > > > > > > > > they > > > > > > > > > > > > > > >>> should > > > > > > > > > > > > > > >>>> use > > > > > > > > > > > > > > >>>>>> and > > > > > > > > > > > > > > >>>>>>> which > > > > > > > > > > > > > > >>>>>>> is fully functional, from that > perspective > > > > > nothing > > > > > > > > blocks > > > > > > > > > > > > marking > > > > > > > > > > > > > > >>> it > > > > > > > > > > > > > > >>>>>>> @Deprecated. > > > > > > > > > > > > > > >>>>>>> As for the remaining work items - there > are > > > > > > primarily > > > > > > > > > three > > > > > > > > > > > > > kinds: > > > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > > > >>>>>>> 1. Where Flink internally uses > > > SourceFunction, > > > > > > > without > > > > > > > > > > > exposing > > > > > > > > > > > > > > >>> this > > > > > > > > > > > > > > >>>>> fact > > > > > > > > > > > > > > >>>>>>> to the > > > > > > > > > > > > > > >>>>>>> outside world: > > > > > > > > > > > > > > >>>>>>> - FLINK-28050 [3] > > > > > > > > > > > > > > >>>>>>> - FLINK-28229 [4] > > > > > > > > > > > > > > >>>>>>> - FLINK-28048 [5] > > > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > > > >>>>>>> 2. Very specific edge cases that might > not > > be > > > > > > covered > > > > > > > > by > > > > > > > > > > the > > > > > > > > > > > > > > >>> Source > > > > > > > > > > > > > > >>>> API > > > > > > > > > > > > > > >>>>>> as > > > > > > > > > > > > > > >>>>>>> is: > > > > > > > > > > > > > > >>>>>>> - FLINK-28054 [6] > > > > > > > > > > > > > > >>>>>>> - FLINK-28051 [7] > > > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > > > >>>>>>> 3. Usability improvements - something > that > > > was > > > > > > easily > > > > > > > > > > doable > > > > > > > > > > > > with > > > > > > > > > > > > > > >>>>>>> SourceFunction, > > > > > > > > > > > > > > >>>>>>> but requires deep knowledge of the > new, > > > > > > > > significantly > > > > > > > > > > > more > > > > > > > > > > > > > > >>>> complex, > > > > > > > > > > > > > > >>>>>>> Source API > > > > > > > > > > > > > > >>>>>>> to achieve: > > > > > > > > > > > > > > >>>>>>> - FLINK-28056 [8] > > > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > > > >>>>>>> In my mind, none of those are blockers > for > > > > > > proceeding > > > > > > > > > with > > > > > > > > > > > > adding > > > > > > > > > > > > > > >>> the > > > > > > > > > > > > > > >>>>>>> @Deprecated > > > > > > > > > > > > > > >>>>>>> annotation: > > > > > > > > > > > > > > >>>>>>> (1) is a simple case of encapsulation, > > > > internals > > > > > > > should > > > > > > > > > not > > > > > > > > > > > > > > >>> concern > > > > > > > > > > > > > > >>>> the > > > > > > > > > > > > > > >>>>>> API > > > > > > > > > > > > > > >>>>>>> users > > > > > > > > > > > > > > >>>>>>> (2) is really only relevant for "exotic" > > use > > > > > cases. > > > > > > > > Does > > > > > > > > > > not > > > > > > > > > > > > mean > > > > > > > > > > > > > > >>> we > > > > > > > > > > > > > > >>>>>> should > > > > > > > > > > > > > > >>>>>>> not > > > > > > > > > > > > > > >>>>>>> consider those, but since it is > irrelevant > > > for > > > > > > 99.9% > > > > > > > of > > > > > > > > > the > > > > > > > > > > > > > > >>> users, I > > > > > > > > > > > > > > >>>> do > > > > > > > > > > > > > > >>>>>> not > > > > > > > > > > > > > > >>>>>>> think > > > > > > > > > > > > > > >>>>>>> we should get stuck here. > > > > > > > > > > > > > > >>>>>>> (3) is purely a nice to have. Formally > > > > speaking, > > > > > > all > > > > > > > of > > > > > > > > > the > > > > > > > > > > > > tools > > > > > > > > > > > > > > >>> are > > > > > > > > > > > > > > >>>>>>> there, it is > > > > > > > > > > > > > > >>>>>>> just that due to the complexity of the > new > > > > Source > > > > > > API > > > > > > > > > some > > > > > > > > > > > > > > >>> "simple" > > > > > > > > > > > > > > >>>>>> things > > > > > > > > > > > > > > >>>>>>> become > > > > > > > > > > > > > > >>>>>>> non-trivial and ideally we want to do > > better > > > > > here. > > > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > > > >>>>>>> [1] > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > https://lists.apache.org/thread/d6cwqw9b3105wcpdkwq7rr4s7x4ywqr9 > > > > > > > > > > > > > > >>>>>>> [2] > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > https://lists.apache.org/thread/kv9rj3w2rmkb8jtss5bqffhw57or7v8v > > > > > > > > > > > > > > >>>>>>> [3] > > > > > > > https://issues.apache.org/jira/browse/FLINK-28050 > > > > > > > > > > > > > > >>>>>>> [4] > > > > > > > https://issues.apache.org/jira/browse/FLINK-28229 > > > > > > > > > > > > > > >>>>>>> [5] > > > > > > > https://issues.apache.org/jira/browse/FLINK-28048 > > > > > > > > > > > > > > >>>>>>> [6] > > > > > > > https://issues.apache.org/jira/browse/FLINK-28054 > > > > > > > > > > > > > > >>>>>>> [7] > > > > > > > https://issues.apache.org/jira/browse/FLINK-28051 > > > > > > > > > > > > > > >>>>>>> [8] > > > > > > > https://issues.apache.org/jira/browse/FLINK-28056 > > > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > > > >>>>>>> On Thu, 29 Jun 2023 at 13:13, Xintong > Song > > < > > > > > > > > > > > > > tonysong...@gmail.com > > > > > > > > > > > > > > >>>>>> wrote: > > > > > > > > > > > > > > >>>>>>>> Thanks for the inputs, Martijn, Jing and > > > Alex. > > > > > > > > > > > > > > >>>>>>>> > > > > > > > > > > > > > > >>>>>>>> @Martijn, > > > > > > > > > > > > > > >>>>>>>> Regarding the Scala supports, I > personally > > > > don't > > > > > > > think > > > > > > > > > "a > > > > > > > > > > > > fully > > > > > > > > > > > > > > >>>>> striked > > > > > > > > > > > > > > >>>>>>>> through experience in the IDE" is > > something > > > we > > > > > > want > > > > > > > to > > > > > > > > > > > avoid, > > > > > > > > > > > > > > >>>>>> especially > > > > > > > > > > > > > > >>>>>>>> given that we are planning to remove the > > > > > > deprecated > > > > > > > > APIs > > > > > > > > > > > soon, > > > > > > > > > > > > > > >>>> unlike > > > > > > > > > > > > > > >>>>>>> when > > > > > > > > > > > > > > >>>>>>>> FLINK-29740 was resolved we didn't > really > > > know > > > > > > when > > > > > > > > they > > > > > > > > > > > would > > > > > > > > > > > > > > >>> be > > > > > > > > > > > > > > >>>>>>> removed. > > > > > > > > > > > > > > >>>>>>>> Moreover, the even entry point for > > > DataStream > > > > > > Scala > > > > > > > > > > > > > > >>>>>>>> (`StreamExecutionEnvironment`) is not > > > > annotated. > > > > > > > > > > > > > > >>>>>>>> > > > > > > > > > > > > > > >>>>>>>> > > > > > > > > > > > > > > >>>>>>>> @Jing and @Alex, > > > > > > > > > > > > > > >>>>>>>> > > > > > > > > > > > > > > >>>>>>>> IIUC, you mean SourceFunction can be > > > annotated > > > > > as > > > > > > > > > > > > `@Deprecated` > > > > > > > > > > > > > > >>> in > > > > > > > > > > > > > > >>>>>> 1.18, > > > > > > > > > > > > > > >>>>>>>> and there's already a PR doing so. > > However, > > > > > after > > > > > > > the > > > > > > > > > > > > > > >>> deprecation, > > > > > > > > > > > > > > >>>>>> there > > > > > > > > > > > > > > >>>>>>>> are still issues that need to be > addressed > > > > > before > > > > > > > > > removing > > > > > > > > > > > it > > > > > > > > > > > > in > > > > > > > > > > > > > > >>>> 2.0? > > > > > > > > > > > > > > >>>>>>> This > > > > > > > > > > > > > > >>>>>>>> sounds a bit weird. If the API cannot be > > > > > dropped, > > > > > > > > which > > > > > > > > > > > means > > > > > > > > > > > > > > >>>> without > > > > > > > > > > > > > > >>>>>>> this > > > > > > > > > > > > > > >>>>>>>> API some of functions cannot be > supported, > > > > then > > > > > > how > > > > > > > > > could > > > > > > > > > > it > > > > > > > > > > > > be > > > > > > > > > > > > > > >>>>>>> deprecated? > > > > > > > > > > > > > > >>>>>>>> How would we expect users to migrate > away > > > from > > > > > it? > > > > > > > > > > > > > > >>>>>>>> > > > > > > > > > > > > > > >>>>>>>> > > > > > > > > > > > > > > >>>>>>>> @Jing, > > > > > > > > > > > > > > >>>>>>>> > > > > > > > > > > > > > > >>>>>>>> Sounds like it's impractical to > deprecate > > > > > > > SinkFunction > > > > > > > > > in > > > > > > > > > > > > 1.18. > > > > > > > > > > > > > > >>> Any > > > > > > > > > > > > > > >>>>>>>> expectation / plan on when / how it can > be > > > > > > > deprecated > > > > > > > > / > > > > > > > > > > > > removed? > > > > > > > > > > > > > > >>>>>>>> > > > > > > > > > > > > > > >>>>>>>> > > > > > > > > > > > > > > >>>>>>>> Best, > > > > > > > > > > > > > > >>>>>>>> > > > > > > > > > > > > > > >>>>>>>> Xintong > > > > > > > > > > > > > > >>>>>>>> > > > > > > > > > > > > > > >>>>>>>> > > > > > > > > > > > > > > >>>>>>>> > > > > > > > > > > > > > > >>>>>>>> On Thu, Jun 29, 2023 at 6:12 PM > Alexander > > > > > Fedulov > > > > > > < > > > > > > > > > > > > > > >>>>>>>> alexander.fedu...@gmail.com> wrote: > > > > > > > > > > > > > > >>>>>>>> > > > > > > > > > > > > > > >>>>>>>>> Hi Xintong, > > > > > > > > > > > > > > >>>>>>>>> > > > > > > > > > > > > > > >>>>>>>>> Thanks for bringing up this topic. I > can > > > > > provide > > > > > > > some > > > > > > > > > > > details > > > > > > > > > > > > > > >>>>>> regarding > > > > > > > > > > > > > > >>>>>>>>> the SourceFunction deprecation efforts. > > > > Marking > > > > > > > > > > > > > > >>> SourceFunction as > > > > > > > > > > > > > > >>>>>>>>> deprecated was not possible until now > > since > > > > we > > > > > > have > > > > > > > > > > > stringent > > > > > > > > > > > > > > >>>>>> compiler > > > > > > > > > > > > > > >>>>>>>>> checks in flink-examples against using > > any > > > > > > > deprecated > > > > > > > > > > APIs. > > > > > > > > > > > > We > > > > > > > > > > > > > > >>>>>> actually > > > > > > > > > > > > > > >>>>>>>>> merged the migration of all examples to > > the > > > > new > > > > > > > > > > > FLIP-27-based > > > > > > > > > > > > > > >>>>>>>>> DataGeneratorSource [1] just two days > ago > > > > [2]. > > > > > > Now > > > > > > > > the > > > > > > > > > PR > > > > > > > > > > > > > > >>> marking > > > > > > > > > > > > > > >>>>>>>>> it @Deprecated is finally unblocked [3] > > (I > > > > > would > > > > > > be > > > > > > > > > > > grateful > > > > > > > > > > > > > > >>> if > > > > > > > > > > > > > > >>>> you > > > > > > > > > > > > > > >>>>>>> could > > > > > > > > > > > > > > >>>>>>>>> merge it). > > > > > > > > > > > > > > >>>>>>>>> > > > > > > > > > > > > > > >>>>>>>>> With regards to the Flink 2.0 scope, I > > > > > compiled a > > > > > > > > list > > > > > > > > > of > > > > > > > > > > > > > > >>> items > > > > > > > > > > > > > > >>>>>>> required > > > > > > > > > > > > > > >>>>>>>> to > > > > > > > > > > > > > > >>>>>>>>> be able to drop the SourceFunction API > > [4] > > > a > > > > > > while > > > > > > > > ago > > > > > > > > > > and > > > > > > > > > > > as > > > > > > > > > > > > > > >>> you > > > > > > > > > > > > > > >>>>> can > > > > > > > > > > > > > > >>>>>>>> see, > > > > > > > > > > > > > > >>>>>>>>> there is still quite some work to be > > done. > > > > Some > > > > > > > items > > > > > > > > > [5] > > > > > > > > > > > > > > >>> might > > > > > > > > > > > > > > >>>>> even > > > > > > > > > > > > > > >>>>>>>>> require additions to the new Source > API. > > > > > > Overall, I > > > > > > > > am > > > > > > > > > > > happy > > > > > > > > > > > > > > >>> to > > > > > > > > > > > > > > >>>>> take > > > > > > > > > > > > > > >>>>>>>>> ownership of completing this work > > package. > > > > > > > > > > > > > > >>>>>>>>> > > > > > > > > > > > > > > >>>>>>>>> Best, > > > > > > > > > > > > > > >>>>>>>>> Alex > > > > > > > > > > > > > > >>>>>>>>> > > > > > > > > > > > > > > >>>>>>>>> > > > > > > > > > > > > > > >>>>>>>>> [1] > > > > > https://cwiki.apache.org/confluence/x/9Av1D > > > > > > > > > > > > > > >>>>>>>>> [2] > > > > https://github.com/apache/flink/pull/21774 > > > > > > > > > > > > > > >>>>>>>>> [3] > > > > https://github.com/apache/flink/pull/20049 > > > > > > > > > > > > > > >>>>>>>>> [4] > > > > > > > > https://issues.apache.org/jira/browse/FLINK-28045 > > > > > > > > > > > > > > >>>>>>>>> [5] > > > > > > > > https://issues.apache.org/jira/browse/FLINK-28054 > > > > > > > > > > > > > > >>>>>>>>> > > > > > > > > > > > > > > >>>>>>>>> On Thu, 29 Jun 2023 at 10:45, Martijn > > > Visser > > > > < > > > > > > > > > > > > > > >>>>>> martijnvis...@apache.org > > > > > > > > > > > > > > >>>>>>>>> wrote: > > > > > > > > > > > > > > >>>>>>>>> > > > > > > > > > > > > > > >>>>>>>>>> Hi Xintong, > > > > > > > > > > > > > > >>>>>>>>>> > > > > > > > > > > > > > > >>>>>>>>>> With regards to the deprecation of the > > > Scala > > > > > > APIs, > > > > > > > > > > during > > > > > > > > > > > > > > >>> the > > > > > > > > > > > > > > >>>> PR > > > > > > > > > > > > > > >>>>>>>>>> review it was requested to not mark > all > > > APIs > > > > > as > > > > > > > > > > deprecated > > > > > > > > > > > > > > >>> but > > > > > > > > > > > > > > >>>>> only > > > > > > > > > > > > > > >>>>>>>>>> the entry point [1], to avoid a fully > > > > striked > > > > > > > > through > > > > > > > > > > > > > > >>>> experience > > > > > > > > > > > > > > >>>>> in > > > > > > > > > > > > > > >>>>>>>>>> the IDE. I think the same idea was > > > > applicable > > > > > on > > > > > > > the > > > > > > > > > > > DataSet > > > > > > > > > > > > > > >>>>> API. I > > > > > > > > > > > > > > >>>>>>>>>> think it depends on how formal we want > > to > > > > > treat > > > > > > > > this: > > > > > > > > > if > > > > > > > > > > > > > > >>> really > > > > > > > > > > > > > > >>>>>>>>>> formal, then we should deprecate them > in > > > > > 1.18. I > > > > > > > > think > > > > > > > > > > in > > > > > > > > > > > > > > >>> both > > > > > > > > > > > > > > >>>>>> cases, > > > > > > > > > > > > > > >>>>>>>>>> it's quite well known that they are > > > > > deprecated. > > > > > > > I'm > > > > > > > > +0 > > > > > > > > > > for > > > > > > > > > > > > > > >>>> either > > > > > > > > > > > > > > >>>>>>> way, > > > > > > > > > > > > > > >>>>>>>>>> as long as we're all agreeing that > they > > > can > > > > be > > > > > > > > removed > > > > > > > > > > in > > > > > > > > > > > > > > >>> 2.0. > > > > > > > > > > > > > > >>>>>>>>>> With regards to Queryable State and > > > > > > > > > Source/SinkFunction, > > > > > > > > > > > +1 > > > > > > > > > > > > > > >>> to > > > > > > > > > > > > > > >>>>> mark > > > > > > > > > > > > > > >>>>>>>>>> these as deprecated. > > > > > > > > > > > > > > >>>>>>>>>> > > > > > > > > > > > > > > >>>>>>>>>> Best regards, > > > > > > > > > > > > > > >>>>>>>>>> > > > > > > > > > > > > > > >>>>>>>>>> Martijn > > > > > > > > > > > > > > >>>>>>>>>> > > > > > > > > > > > > > > >>>>>>>>>> [1] > > > > > > > > > > > > > > >>>>>>>>>> > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/apache/flink/pull/21176#pullrequestreview-1159706808 > > > > > > > > > > > > > > >>>>>>>>>> On Thu, Jun 29, 2023 at 10:23 AM > Xintong > > > > Song > > > > > < > > > > > > > > > > > > > > >>>>>> tonysong...@gmail.com > > > > > > > > > > > > > > >>>>>>>>>> wrote: > > > > > > > > > > > > > > >>>>>>>>>>> Hi devs, > > > > > > > > > > > > > > >>>>>>>>>>> > > > > > > > > > > > > > > >>>>>>>>>>> Looking at the release 2.0 proposals > > > [1], I > > > > > > > noticed > > > > > > > > > > that > > > > > > > > > > > > > > >>> many > > > > > > > > > > > > > > >>>>>> APIs > > > > > > > > > > > > > > >>>>>>>> that > > > > > > > > > > > > > > >>>>>>>>>> are > > > > > > > > > > > > > > >>>>>>>>>>> proposed to be removed in 2.0 are not > > > > (fully) > > > > > > > > > > deprecated > > > > > > > > > > > > > > >>> yet. > > > > > > > > > > > > > > >>>>> We > > > > > > > > > > > > > > >>>>>>>> might > > > > > > > > > > > > > > >>>>>>>>>> want > > > > > > > > > > > > > > >>>>>>>>>>> to properly mark them as > `@Deprecated` > > in > > > > > 1.18 > > > > > > if > > > > > > > > we > > > > > > > > > > > agree > > > > > > > > > > > > > > >>>> they > > > > > > > > > > > > > > >>>>>>>> should > > > > > > > > > > > > > > >>>>>>>>> be > > > > > > > > > > > > > > >>>>>>>>>>> removed in 2.0. Moreover, according > to > > > > > FLIP-321 > > > > > > > [2] > > > > > > > > > > (not > > > > > > > > > > > > > > >>>> voted > > > > > > > > > > > > > > >>>>>> yet > > > > > > > > > > > > > > >>>>>>>> but > > > > > > > > > > > > > > >>>>>>>>>> IMO > > > > > > > > > > > > > > >>>>>>>>>>> is close to consensus IMO), a > migration > > > > > period > > > > > > is > > > > > > > > > > > required > > > > > > > > > > > > > > >>>>> after > > > > > > > > > > > > > > >>>>>>> APIs > > > > > > > > > > > > > > >>>>>>>>> are > > > > > > > > > > > > > > >>>>>>>>>>> deprecated and before they can be > > > removed. > > > > > > > > > > > > > > >>>>>>>>>>> > > > > > > > > > > > > > > >>>>>>>>>>> I might not be familiar with the > status > > > of > > > > > all > > > > > > > the > > > > > > > > > APIs > > > > > > > > > > > > > > >>>> below. > > > > > > > > > > > > > > >>>>> So > > > > > > > > > > > > > > >>>>>>> I'd > > > > > > > > > > > > > > >>>>>>>>>> like > > > > > > > > > > > > > > >>>>>>>>>>> to bring them up and see if there's > any > > > > > concern > > > > > > > > > > regarding > > > > > > > > > > > > > > >>>>>>> deprecating > > > > > > > > > > > > > > >>>>>>>>>> them > > > > > > > > > > > > > > >>>>>>>>>>> in 1.18. If there's concern for > > > deprecating > > > > > > API, > > > > > > > we > > > > > > > > > can > > > > > > > > > > > > > > >>>> start a > > > > > > > > > > > > > > >>>>>>>>> separate > > > > > > > > > > > > > > >>>>>>>>>>> discussion thread for it. For those > > with > > > no > > > > > > > > > objections, > > > > > > > > > > > > > > >>> I'd > > > > > > > > > > > > > > >>>>>> create > > > > > > > > > > > > > > >>>>>>>> JIRA > > > > > > > > > > > > > > >>>>>>>>>>> tickets and try to properly deprecate > > > them > > > > in > > > > > > > 1.18. > > > > > > > > > > > > > > >>>>>>>>>>> > > > > > > > > > > > > > > >>>>>>>>>>> 1. DataSet API > > > > > > > > > > > > > > >>>>>>>>>>> It's described as "legacy", "soft > > > > deprecated" > > > > > > in > > > > > > > > user > > > > > > > > > > > > > > >>>>>> documentation > > > > > > > > > > > > > > >>>>>>>>> [3]. > > > > > > > > > > > > > > >>>>>>>>>>> However, it's not annotated with > > > > > `@Deprecated` > > > > > > in > > > > > > > > > > codes. > > > > > > > > > > > > > > >>>>>> According > > > > > > > > > > > > > > >>>>>>> to > > > > > > > > > > > > > > >>>>>>>>>>> FLIP-131 [4], DataSet API should be > > > > > deprecated > > > > > > > when > > > > > > > > > > > > > > >>>> DataStream > > > > > > > > > > > > > > >>>>>> API > > > > > > > > > > > > > > >>>>>>>> and > > > > > > > > > > > > > > >>>>>>>>>>> Table API / SQL meet certain > > > requirements. > > > > > > > AFAICS, > > > > > > > > > all > > > > > > > > > > > the > > > > > > > > > > > > > > >>>>>>>> requirements > > > > > > > > > > > > > > >>>>>>>>>>> mentioned in the FLIP are already > > > > fulfilled. > > > > > We > > > > > > > > > should > > > > > > > > > > > > > > >>>> annotate > > > > > > > > > > > > > > >>>>>> it > > > > > > > > > > > > > > >>>>>>> as > > > > > > > > > > > > > > >>>>>>>>>>> `@Deprecated` now. > > > > > > > > > > > > > > >>>>>>>>>>> > > > > > > > > > > > > > > >>>>>>>>>>> 2. SourceFunction / SinkFunction > > > > > > > > > > > > > > >>>>>>>>>>> They are described as deprecated in > the > > > > > > > roadmap[5], > > > > > > > > > > and I > > > > > > > > > > > > > > >>>> don't > > > > > > > > > > > > > > >>>>>>> find > > > > > > > > > > > > > > >>>>>>>>>>> anything regarding them in user > > > > > documentation. > > > > > > > But > > > > > > > > > they > > > > > > > > > > > > > > >>> are > > > > > > > > > > > > > > >>>>> also > > > > > > > > > > > > > > >>>>>>> not > > > > > > > > > > > > > > >>>>>>>>>>> annotated with `@Deprecated` in > codes. > > > TBH, > > > > > I'm > > > > > > > not > > > > > > > > > > aware > > > > > > > > > > > > > > >>> of > > > > > > > > > > > > > > >>>>> any > > > > > > > > > > > > > > >>>>>>>> formal > > > > > > > > > > > > > > >>>>>>>>>>> decision to deprecate these. AFAICS, > > the > > > > > > > > replacement > > > > > > > > > > for > > > > > > > > > > > > > > >>>>>>>> SourceFunction > > > > > > > > > > > > > > >>>>>>>>>>> (Source) has already been promoted to > > > > > > `@Public`, > > > > > > > > > while > > > > > > > > > > > the > > > > > > > > > > > > > > >>>>>>>> replacement > > > > > > > > > > > > > > >>>>>>>>>> for > > > > > > > > > > > > > > >>>>>>>>>>> SinkFunction (SinkV2) is still > > > > > > > `@PublicEvolving`. I > > > > > > > > > > found > > > > > > > > > > > > > > >>> a > > > > > > > > > > > > > > >>>>>>>>> discussion[6] > > > > > > > > > > > > > > >>>>>>>>>>> regarding promoting SinkV2 to > > `@Public`, > > > > but > > > > > > it's > > > > > > > > > > unclear > > > > > > > > > > > > > > >>> to > > > > > > > > > > > > > > >>>> me > > > > > > > > > > > > > > >>>>>>> what > > > > > > > > > > > > > > >>>>>>>>> the > > > > > > > > > > > > > > >>>>>>>>>>> conclusion is. > > > > > > > > > > > > > > >>>>>>>>>>> > > > > > > > > > > > > > > >>>>>>>>>>> 3. Queryable State > > > > > > > > > > > > > > >>>>>>>>>>> It's described as approaching > > end-of-life > > > > in > > > > > > the > > > > > > > > > > roadmap > > > > > > > > > > > > > > >>> [5], > > > > > > > > > > > > > > >>>>> but > > > > > > > > > > > > > > >>>>>>> is > > > > > > > > > > > > > > >>>>>>>>>>> neither deprecated in codes nor in > user > > > > > > > > documentation > > > > > > > > > > > > > > >>> [7]. I > > > > > > > > > > > > > > >>>>> also > > > > > > > > > > > > > > >>>>>>>>> found a > > > > > > > > > > > > > > >>>>>>>>>>> discussion [8] about rescuing it from > > > > > > > deprecation, > > > > > > > > > and > > > > > > > > > > it > > > > > > > > > > > > > > >>>> seems > > > > > > > > > > > > > > >>>>>> to > > > > > > > > > > > > > > >>>>>>> me > > > > > > > > > > > > > > >>>>>>>>>> there > > > > > > > > > > > > > > >>>>>>>>>>> are more negative opinions than > > positive > > > > > ones. > > > > > > > > > > > > > > >>>>>>>>>>> > > > > > > > > > > > > > > >>>>>>>>>>> 4. All Scala APIs > > > > > > > > > > > > > > >>>>>>>>>>> I think we agreed to drop Scala API > > > support > > > > > in > > > > > > > > > FLIP-265 > > > > > > > > > > > > > > >>> [9], > > > > > > > > > > > > > > >>>>> and > > > > > > > > > > > > > > >>>>>>> have > > > > > > > > > > > > > > >>>>>>>>>> tried > > > > > > > > > > > > > > >>>>>>>>>>> to deprecate them in FLINK-29740 > [10]. > > > > Also, > > > > > > both > > > > > > > > > user > > > > > > > > > > > > > > >>>>>>> documentation > > > > > > > > > > > > > > >>>>>>>>> and > > > > > > > > > > > > > > >>>>>>>>>>> roadmap[5] shows that scala API > > supports > > > > are > > > > > > > > > > deprecated. > > > > > > > > > > > > > > >>>>> However, > > > > > > > > > > > > > > >>>>>>>>> AFAICS, > > > > > > > > > > > > > > >>>>>>>>>>> none of the APIs in > > > `flink-streaming-scala` > > > > > are > > > > > > > > > > annotated > > > > > > > > > > > > > > >>>> with > > > > > > > > > > > > > > >>>>>>>>>>> `@Deprecated`, and only > > > > > `ExecutionEnvironment` > > > > > > > and > > > > > > > > > > > > > > >>> `package` > > > > > > > > > > > > > > >>>>> are > > > > > > > > > > > > > > >>>>>>>> marked > > > > > > > > > > > > > > >>>>>>>>>>> `@Deprecated` in `flink-scala`. > > > > > > > > > > > > > > >>>>>>>>>>> > > > > > > > > > > > > > > >>>>>>>>>>> Looking forward to your feedback. > > > > > > > > > > > > > > >>>>>>>>>>> > > > > > > > > > > > > > > >>>>>>>>>>> Best, > > > > > > > > > > > > > > >>>>>>>>>>> > > > > > > > > > > > > > > >>>>>>>>>>> Xintong > > > > > > > > > > > > > > >>>>>>>>>>> > > > > > > > > > > > > > > >>>>>>>>>>> > > > > > > > > > > > > > > >>>>>>>>>>> [1] > > > > > > > > > > > > > > >>>>>> > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/2.0+Release > > > > > > > > > > > > > > >>>>>>>>>>> [2] > > > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > > > > > > > https://lists.apache.org/thread/vmhzv8fcw2b33pqxp43486owrxbkd5x9 > > > > > > > > > > > > > > >>>>>>>>>>> [3] > > > > > > > > > > > > > > >>>>>>>>>>> > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://nightlies.apache.org/flink/flink-docs-master/docs/dev/dataset/overview/ > > > > > > > > > > > > > > >>>>>>>>>>> [4] > > > > > > > > > > > > > > >>>>>>>>>>> > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=158866741 > > > > > > > > > > > > > > >>>>>>>>>>> [5] > https://flink.apache.org/roadmap/ > > > > > > > > > > > > > > >>>>>>>>>>> > > > > > > > > > > > > > > >>>>>>>>>>> [6] > > > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > > > > > > > https://lists.apache.org/thread/q62nj89rrz0t5xtggy5n65on95f2rmmx > > > > > > > > > > > > > > >>>>>>>>>>> [7] > > > > > > > > > > > > > > >>>>>>>>>>> > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/fault-tolerance/queryable_state/ > > > > > > > > > > > > > > >>>>>>>>>>> [8] > > > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > > > > > > > https://lists.apache.org/thread/9hmwcjb3q5c24pk3qshjvybfqk62v17m > > > > > > > > > > > > > > >>>>>>>>>>> [9] > > > > > > > > > > > > > > >>>>>>>>>>> > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-265+Deprecate+and+remove+Scala+API+support > > > > > > > > > > > > > > >>>>>>>>>>> [10] > > > > > > > > > https://issues.apache.org/jira/browse/FLINK-29740 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
