Thanks for the input, Jing. I'd also be +1 for option 1. Best,
Xintong On Mon, Jul 3, 2023 at 7:20 PM Jing Ge <j...@ververica.com.invalid> wrote: > Hi Xingtong, > > Option 1, secure plan would be: > > 1. graduate kafka, File, JDBC connectors to @Public > 2. graduate SinkV2 to @Public > 3. remove SinkFunction. > > Option 2, risky plan but at a fast pace: > > 1. graduate SinkV2 to @Public and expecting more maintenance effort since > there are many known and unsolved issues. > 2. remove SinkFunction. > 3. It depends on the connectors' contributors whether connectors can > upgrade to Flink 2.0, since we moved forward with SinkV2 API without taking > care of implementations in external connectors. > > I am ok with both of them and personally prefer option 1. > > Best regards, > Jing > > > On Fri, Jun 30, 2023 at 3:41 AM Xintong Song <tonysong...@gmail.com> > wrote: > > > I see. Thanks for the explanation. I may have not looked into this deeply > > enough, and would trust the decision from you and the community members > who > > participated in the discussion & vote. > > > > Best, > > > > Xintong > > > > > > > > On Thu, Jun 29, 2023 at 10:28 PM Alexander Fedulov < > > alexander.fedu...@gmail.com> wrote: > > > > > > However, I'm not sure about 2. > > > > > > I am not aware of a bylaw that states the specific requirements in > order > > to > > > mark something as @Deprecated. My understanding from the discussion and > > the > > > vote was that the community recognizes the necessity to make it > explicit > > > that > > > the usage of the SourceFunction API is discouraged. This can actually > > > stimulate > > > authors of connectors that rely on this very specific and non-baseline > > > functionality to contribute extensions to the new Source API themselves > > in > > > order to > > > close the gap. ExternallyInducedSource, for instance, was driven by > > Pravega > > > to > > > begin with, since it was only needed for their purposes [1]. We are not > > > removing > > > anything - until 2.0 everything will continue to work and we can work > on > > > resolving the limitations until then, I personally don't see a big > issue > > > here. > > > > > > >Do you think it is feasible to resolve them by the feature freeze date > > of > > > 1.18? > > > No, these are rather complex additions that would probably require > > FLIP(s). > > > > > > [1] > > > > > > > > > https://flink.apache.org/2022/01/20/pravega-flink-connector-101/#checkpoint-integration > > > > > > On Thu, 29 Jun 2023 at 14:25, Xintong Song <tonysong...@gmail.com> > > wrote: > > > > > > > Thanks for the explanation, Alex. > > > > > > > > Not blocking the deprecation on 1 & 3 makes sense to me. However, I'm > > not > > > > sure about 2. > > > > > > > > It sounds to me that, without FLINK-28051 & FLINK-28054, some of the > > > > connectors cannot migrate to the new Source API, or at least further > > > > investigation is needed to understand the situation. If this is the > > case, > > > > we probably should not deprecate the API until these issues are > > resolved. > > > > Do you think it is feasible to resolve them by the feature freeze > date > > of > > > > 1.18? > > > > > > > > Best, > > > > > > > > Xintong > > > > > > > > > > > > > > > > On Thu, Jun 29, 2023 at 8:02 PM Alexander Fedulov < > > > > alexander.fedu...@gmail.com> wrote: > > > > > > > > > @Xintong > > > > > The original discussion [1] and vote [2] converged on the idea that > > it > > > is > > > > > better > > > > > to make it clear to the users that they should stop using > > > SourceFunction > > > > > since it > > > > > is going away. The longer we do not have this indication, the more > > user > > > > > implementations will be based on it and the more pain will be > induced > > > > when > > > > > we > > > > > finally drop it. Users now have an alternative API that they should > > use > > > > and > > > > > which > > > > > is fully functional, from that perspective nothing blocks marking > it > > > > > @Deprecated. > > > > > As for the remaining work items - there are primarily three kinds: > > > > > > > > > > 1. Where Flink internally uses SourceFunction, without exposing > this > > > fact > > > > > to the > > > > > outside world: > > > > > - FLINK-28050 [3] > > > > > - FLINK-28229 [4] > > > > > - FLINK-28048 [5] > > > > > > > > > > 2. Very specific edge cases that might not be covered by the Source > > API > > > > as > > > > > is: > > > > > - FLINK-28054 [6] > > > > > - FLINK-28051 [7] > > > > > > > > > > 3. Usability improvements - something that was easily doable with > > > > > SourceFunction, > > > > > but requires deep knowledge of the new, significantly more > > complex, > > > > > Source API > > > > > to achieve: > > > > > - FLINK-28056 [8] > > > > > > > > > > In my mind, none of those are blockers for proceeding with adding > the > > > > > @Deprecated > > > > > annotation: > > > > > (1) is a simple case of encapsulation, internals should not concern > > the > > > > API > > > > > users > > > > > (2) is really only relevant for "exotic" use cases. Does not mean > we > > > > should > > > > > not > > > > > consider those, but since it is irrelevant for 99.9% of the users, > I > > do > > > > not > > > > > think > > > > > we should get stuck here. > > > > > (3) is purely a nice to have. Formally speaking, all of the tools > are > > > > > there, it is > > > > > just that due to the complexity of the new Source API some "simple" > > > > things > > > > > become > > > > > non-trivial and ideally we want to do better here. > > > > > > > > > > [1] > https://lists.apache.org/thread/d6cwqw9b3105wcpdkwq7rr4s7x4ywqr9 > > > > > [2] > https://lists.apache.org/thread/kv9rj3w2rmkb8jtss5bqffhw57or7v8v > > > > > [3] https://issues.apache.org/jira/browse/FLINK-28050 > > > > > [4] https://issues.apache.org/jira/browse/FLINK-28229 > > > > > [5] https://issues.apache.org/jira/browse/FLINK-28048 > > > > > [6] https://issues.apache.org/jira/browse/FLINK-28054 > > > > > [7] https://issues.apache.org/jira/browse/FLINK-28051 > > > > > [8] https://issues.apache.org/jira/browse/FLINK-28056 > > > > > > > > > > On Thu, 29 Jun 2023 at 13:13, Xintong Song <tonysong...@gmail.com> > > > > wrote: > > > > > > > > > > > Thanks for the inputs, Martijn, Jing and Alex. > > > > > > > > > > > > @Martijn, > > > > > > Regarding the Scala supports, I personally don't think "a fully > > > striked > > > > > > through experience in the IDE" is something we want to avoid, > > > > especially > > > > > > given that we are planning to remove the deprecated APIs soon, > > unlike > > > > > when > > > > > > FLINK-29740 was resolved we didn't really know when they would be > > > > > removed. > > > > > > Moreover, the even entry point for DataStream Scala > > > > > > (`StreamExecutionEnvironment`) is not annotated. > > > > > > > > > > > > > > > > > > @Jing and @Alex, > > > > > > > > > > > > IIUC, you mean SourceFunction can be annotated as `@Deprecated` > in > > > > 1.18, > > > > > > and there's already a PR doing so. However, after the > deprecation, > > > > there > > > > > > are still issues that need to be addressed before removing it in > > 2.0? > > > > > This > > > > > > sounds a bit weird. If the API cannot be dropped, which means > > without > > > > > this > > > > > > API some of functions cannot be supported, then how could it be > > > > > deprecated? > > > > > > How would we expect users to migrate away from it? > > > > > > > > > > > > > > > > > > @Jing, > > > > > > > > > > > > Sounds like it's impractical to deprecate SinkFunction in 1.18. > Any > > > > > > expectation / plan on when / how it can be deprecated / removed? > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > > > > Xintong > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Jun 29, 2023 at 6:12 PM Alexander Fedulov < > > > > > > alexander.fedu...@gmail.com> wrote: > > > > > > > > > > > > > Hi Xintong, > > > > > > > > > > > > > > Thanks for bringing up this topic. I can provide some details > > > > regarding > > > > > > > the SourceFunction deprecation efforts. Marking SourceFunction > as > > > > > > > deprecated was not possible until now since we have stringent > > > > compiler > > > > > > > checks in flink-examples against using any deprecated APIs. We > > > > actually > > > > > > > merged the migration of all examples to the new FLIP-27-based > > > > > > > DataGeneratorSource [1] just two days ago [2]. Now the PR > marking > > > > > > > it @Deprecated is finally unblocked [3] (I would be grateful if > > you > > > > > could > > > > > > > merge it). > > > > > > > > > > > > > > With regards to the Flink 2.0 scope, I compiled a list of items > > > > > required > > > > > > to > > > > > > > be able to drop the SourceFunction API [4] a while ago and as > you > > > can > > > > > > see, > > > > > > > there is still quite some work to be done. Some items [5] might > > > even > > > > > > > require additions to the new Source API. Overall, I am happy to > > > take > > > > > > > ownership of completing this work package. > > > > > > > > > > > > > > Best, > > > > > > > Alex > > > > > > > > > > > > > > > > > > > > > [1] https://cwiki.apache.org/confluence/x/9Av1D > > > > > > > [2] https://github.com/apache/flink/pull/21774 > > > > > > > [3] https://github.com/apache/flink/pull/20049 > > > > > > > [4] https://issues.apache.org/jira/browse/FLINK-28045 > > > > > > > [5] https://issues.apache.org/jira/browse/FLINK-28054 > > > > > > > > > > > > > > On Thu, 29 Jun 2023 at 10:45, Martijn Visser < > > > > martijnvis...@apache.org > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > Hi Xintong, > > > > > > > > > > > > > > > > With regards to the deprecation of the Scala APIs, during the > > PR > > > > > > > > review it was requested to not mark all APIs as deprecated > but > > > only > > > > > > > > the entry point [1], to avoid a fully striked through > > experience > > > in > > > > > > > > the IDE. I think the same idea was applicable on the DataSet > > > API. I > > > > > > > > think it depends on how formal we want to treat this: if > really > > > > > > > > formal, then we should deprecate them in 1.18. I think in > both > > > > cases, > > > > > > > > it's quite well known that they are deprecated. I'm +0 for > > either > > > > > way, > > > > > > > > as long as we're all agreeing that they can be removed in > 2.0. > > > > > > > > > > > > > > > > With regards to Queryable State and Source/SinkFunction, +1 > to > > > mark > > > > > > > > these as deprecated. > > > > > > > > > > > > > > > > Best regards, > > > > > > > > > > > > > > > > Martijn > > > > > > > > > > > > > > > > [1] > > > > > > > > > > > > > > > > > > > > https://github.com/apache/flink/pull/21176#pullrequestreview-1159706808 > > > > > > > > > > > > > > > > On Thu, Jun 29, 2023 at 10:23 AM Xintong Song < > > > > tonysong...@gmail.com > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > Hi devs, > > > > > > > > > > > > > > > > > > Looking at the release 2.0 proposals [1], I noticed that > many > > > > APIs > > > > > > that > > > > > > > > are > > > > > > > > > proposed to be removed in 2.0 are not (fully) deprecated > yet. > > > We > > > > > > might > > > > > > > > want > > > > > > > > > to properly mark them as `@Deprecated` in 1.18 if we agree > > they > > > > > > should > > > > > > > be > > > > > > > > > removed in 2.0. Moreover, according to FLIP-321 [2] (not > > voted > > > > yet > > > > > > but > > > > > > > > IMO > > > > > > > > > is close to consensus IMO), a migration period is required > > > after > > > > > APIs > > > > > > > are > > > > > > > > > deprecated and before they can be removed. > > > > > > > > > > > > > > > > > > I might not be familiar with the status of all the APIs > > below. > > > So > > > > > I'd > > > > > > > > like > > > > > > > > > to bring them up and see if there's any concern regarding > > > > > deprecating > > > > > > > > them > > > > > > > > > in 1.18. If there's concern for deprecating API, we can > > start a > > > > > > > separate > > > > > > > > > discussion thread for it. For those with no objections, I'd > > > > create > > > > > > JIRA > > > > > > > > > tickets and try to properly deprecate them in 1.18. > > > > > > > > > > > > > > > > > > 1. DataSet API > > > > > > > > > It's described as "legacy", "soft deprecated" in user > > > > documentation > > > > > > > [3]. > > > > > > > > > However, it's not annotated with `@Deprecated` in codes. > > > > According > > > > > to > > > > > > > > > FLIP-131 [4], DataSet API should be deprecated when > > DataStream > > > > API > > > > > > and > > > > > > > > > Table API / SQL meet certain requirements. AFAICS, all the > > > > > > requirements > > > > > > > > > mentioned in the FLIP are already fulfilled. We should > > annotate > > > > it > > > > > as > > > > > > > > > `@Deprecated` now. > > > > > > > > > > > > > > > > > > 2. SourceFunction / SinkFunction > > > > > > > > > They are described as deprecated in the roadmap[5], and I > > don't > > > > > find > > > > > > > > > anything regarding them in user documentation. But they are > > > also > > > > > not > > > > > > > > > annotated with `@Deprecated` in codes. TBH, I'm not aware > of > > > any > > > > > > formal > > > > > > > > > decision to deprecate these. AFAICS, the replacement for > > > > > > SourceFunction > > > > > > > > > (Source) has already been promoted to `@Public`, while the > > > > > > replacement > > > > > > > > for > > > > > > > > > SinkFunction (SinkV2) is still `@PublicEvolving`. I found a > > > > > > > discussion[6] > > > > > > > > > regarding promoting SinkV2 to `@Public`, but it's unclear > to > > me > > > > > what > > > > > > > the > > > > > > > > > conclusion is. > > > > > > > > > > > > > > > > > > 3. Queryable State > > > > > > > > > It's described as approaching end-of-life in the roadmap > [5], > > > but > > > > > is > > > > > > > > > neither deprecated in codes nor in user documentation [7]. > I > > > also > > > > > > > found a > > > > > > > > > discussion [8] about rescuing it from deprecation, and it > > seems > > > > to > > > > > me > > > > > > > > there > > > > > > > > > are more negative opinions than positive ones. > > > > > > > > > > > > > > > > > > 4. All Scala APIs > > > > > > > > > I think we agreed to drop Scala API support in FLIP-265 > [9], > > > and > > > > > have > > > > > > > > tried > > > > > > > > > to deprecate them in FLINK-29740 [10]. Also, both user > > > > > documentation > > > > > > > and > > > > > > > > > roadmap[5] shows that scala API supports are deprecated. > > > However, > > > > > > > AFAICS, > > > > > > > > > none of the APIs in `flink-streaming-scala` are annotated > > with > > > > > > > > > `@Deprecated`, and only `ExecutionEnvironment` and > `package` > > > are > > > > > > marked > > > > > > > > > `@Deprecated` in `flink-scala`. > > > > > > > > > > > > > > > > > > Looking forward to your feedback. > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > > > > > > > > > > Xintong > > > > > > > > > > > > > > > > > > > > > > > > > > > [1] > > > > https://cwiki.apache.org/confluence/display/FLINK/2.0+Release > > > > > > > > > > > > > > > > > > [2] > > > > > https://lists.apache.org/thread/vmhzv8fcw2b33pqxp43486owrxbkd5x9 > > > > > > > > > > > > > > > > > > [3] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://nightlies.apache.org/flink/flink-docs-master/docs/dev/dataset/overview/ > > > > > > > > > > > > > > > > > > [4] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=158866741 > > > > > > > > > > > > > > > > > > [5] https://flink.apache.org/roadmap/ > > > > > > > > > > > > > > > > > > [6] > > > > > https://lists.apache.org/thread/q62nj89rrz0t5xtggy5n65on95f2rmmx > > > > > > > > > > > > > > > > > > [7] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/fault-tolerance/queryable_state/ > > > > > > > > > > > > > > > > > > [8] > > > > > https://lists.apache.org/thread/9hmwcjb3q5c24pk3qshjvybfqk62v17m > > > > > > > > > > > > > > > > > > [9] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-265+Deprecate+and+remove+Scala+API+support > > > > > > > > > > > > > > > > > > [10] https://issues.apache.org/jira/browse/FLINK-29740 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
