+1 for deprecating SourceFunction from me as well. And a big THANK YOU to Alex Fedulov for bringing forward FLIP-238.
David On Fri, Jun 10, 2022 at 4:03 AM Lijie Wang <wangdachui9...@gmail.com> wrote: > Hi all, > > Sorry for my mistake. The `StreamExecutionEnvironment#readFiles` and can be > easily replaced by `FileSource#forRecordStreamFormat/forBulkFileFormat`. I > have no other concerns. > > +1 to deprecate SourceFunction and deprecate the methods (in > StreamExecutionEnvironment) based on SourceFunction . > > Best, > Lijie > > Konstantin Knauf <kna...@apache.org> 于2022年6月10日周五 05:11写道： > > > Hi everyone, > > > > thank you Jing for redirecting the discussion back to the topic at hand. > I > > agree with all of your points. > > > > +1 to deprecate SourceFunction > > > > Is there really no replacement for the > StreamExecutionEnvironment#readXXX. > > There is already a FLIP-27 based FileSource, right? What's missing to > > recommend using that as opposed to the the readXXX methods? > > > > Cheers, > > > > Konstantin > > > > Am Do., 9. Juni 2022 um 20:11 Uhr schrieb Alexander Fedulov < > > alexan...@ververica.com>: > > > > > Hi all, > > > > > > It seems that there is some understandable cautiousness with regard to > > > deprecating methods and subclasses that do not have alternatives just > > yet. > > > > > > We should probably first agree if it is in general OK for Flink to use > > > @Deprecated > > > annotation for parts of the code that do not have alternatives. In that > > > case, > > > we could add a comment along the lines of: > > > "This implementation is based on a deprecated SourceFunction API that > > > will gradually be phased out from Flink. No direct substitute exists at > > the > > > moment. > > > If you want to have a more future-proof solution, consider helping the > > > project by > > > contributing an implementation based on the new Source API." > > > > > > This should clearly communicate the message that usage of these > > > methods/classes > > > is discouraged and at the same time promote contributions for > addressing > > > the gap. > > > What do you think? > > > > > > Best, > > > Alexander Fedulov > > > > > > > > > On Thu, Jun 9, 2022 at 6:27 PM Ingo Bürk <airbla...@apache.org> wrote: > > > > > > > Hi, > > > > > > > > these APIs don't expose the underlying source directly, so I don't > > think > > > > we need to worry about deprecating them as well. There's also nothing > > > > inherently wrong with using a deprecated API internally, though even > > > > just for the experience of using our own new APIs I would personally > > say > > > > that they should be migrated to the new Source API. It's hard to > reason > > > > that users must migrate to a new API if we don't do it internally as > > > well. > > > > > > > > > > > > Best > > > > Ingo > > > > > > > > On 09.06.22 15:41, Lijie Wang wrote: > > > > > Hi Martijn, > > > > > > > > > > I don't mean it's a blocker. Just a information. And I'm also +1 > for > > > > this. > > > > > > > > > > Put it another way: should we migrate the `#readFile(...)` to new > API > > > or > > > > > provide a similar method "readxxx“ based on the new Source API? > > > > > > > > > > And if we don't migrate it, does it mean that the `#readFile(...)` > > > should > > > > > also be marked as deprecated? > > > > > > > > > > Best, > > > > > Lijie > > > > > > > > > > Martijn Visser <martijnvis...@apache.org> 于2022年6月9日周四 21:03写道： > > > > > > > > > >> Hi Lijie, > > > > >> > > > > >> I don't see any problem with deprecating those methods at this > > moment, > > > > as > > > > >> long as we don't remove them until the replacements are available. > > > > Besides > > > > >> that, are we sure there are no replacements already, especially > with > > > the > > > > >> new FileSource? > > > > >> > > > > >> Best regards, > > > > >> > > > > >> Martijn > > > > >> > > > > >> Op do 9 jun. 2022 om 14:23 schreef Lijie Wang < > > > wangdachui9...@gmail.com > > > > >: > > > > >> > > > > >>> Hi all, > > > > >>> > > > > >>> FYI, currently, some commonly used methods in > > > > StreamExecutionEnvironment > > > > >>> are still based on the old SourceFunction (and there is no > > > > alternative): > > > > >>> `StreamExecutionEnvironment#readFile(...)` > > > > >>> `StreamExecutionEnvironment#readTextFile(...)` > > > > >>> > > > > >>> I think these should be migrated to the new source API before > > > deprecate > > > > >> the > > > > >>> SourceFunction. > > > > >>> > > > > >>> Best, > > > > >>> Lijie > > > > >>> > > > > >>> Martijn Visser <martijnvis...@apache.org> 于2022年6月9日周四 16:05写道： > > > > >>> > > > > >>>> Hi all, > > > > >>>> > > > > >>>> I think implicitly we've already considered the SourceFunction > and > > > > >>>> SinkFunction as deprecated. They are even marked as so on the > > Flink > > > > >>> roadmap > > > > >>>> [1]. That also shows that connectors that are using these > > interfaces > > > > >> are > > > > >>>> either approaching end-of-life. The fact that we're actively > > > migrating > > > > >>>> connectors from Source/SinkFunction to FLIP-27/FLIP-143 (plus > > add-on > > > > >>> FLIPs) > > > > >>>> shows that we've already determined that target. > > > > >>>> > > > > >>>> With regards to the motivation of FLIP-27, I think reading up on > > the > > > > >>>> original discussion thread is also worthwhile [2] to see more > > > context. > > > > >>>> FLIP-27 was also very important as it brought a unified > connector > > > > which > > > > >>> can > > > > >>>> support both streaming and batch (with batch being considered a > > > > special > > > > >>>> case of streaming in Flink's vision). > > > > >>>> > > > > >>>> So +1 to deprecate SourceFunction. I would also argue that we > > should > > > > >>>> already mark the SinkFunction as deprecated to avoid having this > > > > >>> discussion > > > > >>>> again in a couple of months. > > > > >>>> > > > > >>>> Best regards, > > > > >>>> > > > > >>>> Martijn > > > > >>>> > > > > >>>> [1] https://flink.apache.org/roadmap.html > > > > >>>> [2] > > > https://lists.apache.org/thread/334co89dbhc8qpr9nvmz8t1gp4sz2c8y > > > > >>>> > > > > >>>> Op do 9 jun. 2022 om 09:48 schreef Jing Ge <j...@ververica.com > >: > > > > >>>> > > > > >>>>> Hi, > > > > >>>>> > > > > >>>>> I am very happy to see opinions from different perspectives. > That > > > > >> will > > > > >>>> help > > > > >>>>> us understand the problem better. Thanks all for the > informative > > > > >>>>> discussion. > > > > >>>>> > > > > >>>>> Let's see the big picture and check following facts together: > > > > >>>>> > > > > >>>>> 1. FLIP-27 was intended to solve some technical issues that are > > > very > > > > >>>>> difficult to solve with SourceFunction[1]. When we say > > > > >> "SourceFunction > > > > >>> is > > > > >>>>> easy", well, it depends. If we take a look at the > implementation > > of > > > > >> the > > > > >>>>> Kafka connector, we will know how complicated it is to build a > > > > >> serious > > > > >>>>> connector for production with the old SourceFunction. To every > > > > >> problem > > > > >>>>> there is a solution and to every solution there is a problem. > The > > > > >> fact > > > > >>> is > > > > >>>>> that there is no perfect but a feasible solution. If we try to > > > solve > > > > >>>>> complicated problems, we have to expose some complexity. > > Comparing > > > to > > > > >>>>> connectors for POC, demo, training(no offense), I would also > > solve > > > > >>> issues > > > > >>>>> for connectors like Kafka connector that are widely used in > > > > >> production > > > > >>>> with > > > > >>>>> higher priority. I think that should be one reason why FLIP-27 > > has > > > > >> been > > > > >>>>> designed and why the new source API went public. > > > > >>>>> > > > > >>>>> 2. FLIP-27 and the implementation was introduced roughly at the > > end > > > > >> of > > > > >>>> 2019 > > > > >>>>> and went public on 19.04.2021, which means Flink has provided > two > > > > >>>> different > > > > >>>>> public/graduated source solutions for more than one year. On > the > > > day > > > > >>> that > > > > >>>>> the new source API went public, there should be a consensus in > > the > > > > >>>>> community that we should start the migration. Old > SourceFunction > > > > >>>> interface, > > > > >>>>> in the ideal case, should have been deprecated on that day, > > > otherwise > > > > >>> we > > > > >>>>> should not graduate the new source API to avoid confusing > > > (connector) > > > > >>>>> developers[2]. > > > > >>>>> > > > > >>>>> 3. It is true that the new source API is hard to understand and > > > even > > > > >>> hard > > > > >>>>> to implement for simple cases. Thanks for the feedback. That is > > > > >>> something > > > > >>>>> we need to improve. The current design&implementation could be > > > > >>> considered > > > > >>>>> as the low level API. The next step is to create the high level > > API > > > > >> to > > > > >>>>> reduce some unnecessary complexity for those simple cases. But, > > > IMHO, > > > > >>>> this > > > > >>>>> should not be the prerequisite to postpone the deprecation of > the > > > old > > > > >>>>> SourceFunction APIs. > > > > >>>>> > > > > >>>>> 4. As long as the old SourceFunction is not marked as > deprecated, > > > > >>>>> developers will continue asking which one should be used. Let's > > > make > > > > >> a > > > > >>>>> concrete example. If a new connector is developed now and the > > > > >> developer > > > > >>>>> asks for a suggestion of the choice between the old and new > > source > > > > >> API > > > > >>> on > > > > >>>>> the ML, which one should we suggest? I think it should be the > new > > > > >>> Source > > > > >>>>> API. If a fresh new connector has been developed with the old > > > > >>>>> SourceFunction API before asking for the consensus in the > > community > > > > >> and > > > > >>>> the > > > > >>>>> developer wants to merge it to the master. Should we allow it? > If > > > the > > > > >>>>> answer of all these questions is pointing to the new Source > API, > > > the > > > > >>> old > > > > >>>>> SourceFunction is de facto already deprecated, just has not > been > > > > >> marked > > > > >>>> as > > > > >>>>> @deprecated, which confuses developers even more. > > > > >>>>> > > > > >>>>> Best regards, > > > > >>>>> Jing > > > > >>>>> > > > > >>>>> [1] > > > > >>>>> > > > > >>>>> > > > > >>>> > > > > >>> > > > > >> > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-27%3A+Refactor+Source+Interface > > > > >>>>> [2] > > > https://lists.apache.org/thread/7okp4y46n3o3rx5mn0t3qobrof8zxwqs > > > > >>>>> > > > > >>>>> On Wed, Jun 8, 2022 at 2:21 AM Alexander Fedulov < > > > > >>>> alexan...@ververica.com> > > > > >>>>> wrote: > > > > >>>>> > > > > >>>>>> Hey Austin, > > > > >>>>>> > > > > >>>>>> Since we are getting deeper into the implementation details of > > the > > > > >>>>>> DataGeneratorSource > > > > >>>>>> and it is not the main topic of this thread, I propose to move > > our > > > > >>>>>> discussion to where it belongs: [DISCUSS] FLIP-238 [1]. Could > > you > > > > >>>> please > > > > >>>>>> briefly formulate your requirements to make it easier for the > > > > >> others > > > > >>> to > > > > >>>>>> follow? I am happy to continue this conversation there. > > > > >>>>>> > > > > >>>>>> [1] > > > > >> https://lists.apache.org/thread/7gjxto1rmkpff4kl54j8nlg5db2rqhkt > > > > >>>>>> > > > > >>>>>> Best, > > > > >>>>>> Alexander Fedulov > > > > >>>>>> > > > > >>>>>> On Tue, Jun 7, 2022 at 6:14 PM Austin Cawley-Edwards < > > > > >>>>>> austin.caw...@gmail.com> wrote: > > > > >>>>>> > > > > >>>>>>>> @Austin, in the FLIP I mentioned above [1], the user is > > > > >> expected > > > > >>> to > > > > >>>>>>> pass a MapFunction<Long, > > > > >>>>>>> OUT> > > > > >>>>>>> to the generator. I wonder if you could have your external > > client > > > > >>> and > > > > >>>>>>> polling logic wrapped in a custom > > > > >>>>>>> MapFunction implementation class? Would that answer your > needs > > or > > > > >>> do > > > > >>>>> you > > > > >>>>>>> have some > > > > >>>>>>> more sophisticated scenario in mind? > > > > >>>>>>> > > > > >>>>>>> At first glance, the FLIP looks good but for this case in > > regards > > > > >>> to > > > > >>>>> the > > > > >>>>>>> map function, but leaves out 1) ability to control polling > > > > >>> intervals > > > > >>>>> and > > > > >>>>>> 2) > > > > >>>>>>> ability to produce an unknown number of records, both > per-poll > > > > >> and > > > > >>>>>> overall > > > > >>>>>>> boundedness. Do you think something like this could be built > > from > > > > >>> the > > > > >>>>>> same > > > > >>>>>>> pieces? > > > > >>>>>>> I'm also wondering what handles threading, is that on the > user > > or > > > > >>> is > > > > >>>>> that > > > > >>>>>>> part of the DataGeneratorSource? > > > > >>>>>>> > > > > >>>>>>> Best, > > > > >>>>>>> Austin > > > > >>>>>>> > > > > >>>>>>> On Tue, Jun 7, 2022 at 9:34 AM Alexander Fedulov < > > > > >>>>>> alexan...@ververica.com> > > > > >>>>>>> wrote: > > > > >>>>>>> > > > > >>>>>>>> Hi everyone, > > > > >>>>>>>> > > > > >>>>>>>> Thanks for all the input and a lively discussion. It seems > > that > > > > >>>> there > > > > >>>>>> is > > > > >>>>>>> a > > > > >>>>>>>> consensus that due to > > > > >>>>>>>> the inherent complexity of FLIP-27 sources we should provide > > > > >> more > > > > >>>>>>>> user-facing utilities to bridge > > > > >>>>>>>> the gap between the existing SourceFunction-based > > functionality > > > > >>> and > > > > >>>>> the > > > > >>>>>>> new > > > > >>>>>>>> APIs. > > > > >>>>>>>> > > > > >>>>>>>> To start addressing this I picked the issue that David > raised > > > > >> and > > > > >>>>> many > > > > >>>>>>>> upvoted. Here is a proposal > > > > >>>>>>>> for the new DataGeneratorSource: FLIP-238 [1]. Please take > a > > > > >>>> look, I > > > > >>>>>> am > > > > >>>>>>>> going to open a separate > > > > >>>>>>>> discussion thread on it shortly. > > > > >>>>>>>> > > > > >>>>>>>> Jing also raised some great points regarding the interfaces > > and > > > > >>>>>>> subclasses. > > > > >>>>>>>> It seems to me that > > > > >>>>>>>> what might actually help is some sort of a "soft > deprecation" > > > > >>>> concept > > > > >>>>>> and > > > > >>>>>>>> annotation. It could be > > > > >>>>>>>> used in places where we do not have an alternative > > > > >> implementation > > > > >>>>> yet, > > > > >>>>>>> but > > > > >>>>>>>> we clearly want > > > > >>>>>>>> to indicate that continuing to build on top of these > > interfaces > > > > >>> is > > > > >>>>>>>> discouraged. The area of > > > > >>>>>>>> impact of deprecating all SourceFunction subclasses is > rather > > > > >>> big, > > > > >>>>> and > > > > >>>>>> we > > > > >>>>>>>> can expect it to > > > > >>>>>>>> take a while. The hope would be that if in the meantime > > someone > > > > >>>> finds > > > > >>>>>>>> themselves using one of > > > > >>>>>>>> such old APIs, the "soft deprecation" annotation will be a > > > > >> clear > > > > >>>>>>> indication > > > > >>>>>>>> and encouragement to > > > > >>>>>>>> work on introducing an alternative FLIP-27-based > > implementation > > > > >>>>>> instead. > > > > >>>>>>>> > > > > >>>>>>>> @Austin, in the FLIP I mentioned above [1], the user is > > > > >> expected > > > > >>> to > > > > >>>>>>>> pass a MapFunction<Long, > > > > >>>>>>>> OUT> > > > > >>>>>>>> to the generator. I wonder if you could have your external > > > > >> client > > > > >>>> and > > > > >>>>>>>> polling logic wrapped in a custom > > > > >>>>>>>> MapFunction implementation class? Would that answer your > needs > > > > >> or > > > > >>>> do > > > > >>>>>> you > > > > >>>>>>>> have some > > > > >>>>>>>> more sophisticated scenario in mind? > > > > >>>>>>>> > > > > >>>>>>>> [1] https://cwiki.apache.org/confluence/x/9Av1D > > > > >>>>>>>> Best, > > > > >>>>>>>> Alexander Fedulov > > > > >>>>>>>> > > > > >>>>>>>> On Mon, Jun 6, 2022 at 7:08 PM Austin Cawley-Edwards < > > > > >>>>>>>> austin.caw...@gmail.com> wrote: > > > > >>>>>>>> > > > > >>>>>>>>> Thanks for the nice discussion all. > > > > >>>>>>>>> > > > > >>>>>>>>> I was recently trying to implement a very simple polling > > > > >> source > > > > >>>> and > > > > >>>>>>>>> would've loved a higher-level base to work from. I'm > > > > >> wondering > > > > >>> if > > > > >>>>> in > > > > >>>>>>>>> addition to the data generator use cases, it would be good > to > > > > >>>>>> support a > > > > >>>>>>>>> simple non-parallel polling abstraction to make it easier > to, > > > > >>> for > > > > >>>>>>>> instance, > > > > >>>>>>>>> start prototyping with data in existing APIs without > adding a > > > > >>>> Kafka > > > > >>>>>> or > > > > >>>>>>>> such > > > > >>>>>>>>> in the middle. > > > > >>>>>>>>> > > > > >>>>>>>>> Best, > > > > >>>>>>>>> Austin > > > > >>>>>>>>> > > > > >>>>>>>>> On Mon, Jun 6, 2022 at 10:02 AM tison < > wander4...@gmail.com> > > > > >>>>> wrote: > > > > >>>>>>>>> > > > > >>>>>>>>>> Well. It's a bit off-topic. For deprecating SourceFunction > > > > >> as > > > > >>>>>> FLIP-27 > > > > >>>>>>>>>> series works go ahead, +1 from my side. It's a significant > > > > >>> work > > > > >>>>>>> towards > > > > >>>>>>>>> the > > > > >>>>>>>>>> unification of batch and streaming effort :) > > > > >>>>>>>>>> > > > > >>>>>>>>>> Best, > > > > >>>>>>>>>> tison. > > > > >>>>>>>>>> > > > > >>>>>>>>>> > > > > >>>>>>>>>> tison <wander4...@gmail.com> 于2022年6月6日周一 21:54写道： > > > > >>>>>>>>>> > > > > >>>>>>>>>>> The starting point of the version bump and removal > > > > >> question > > > > >>>> is > > > > >>>>>> that > > > > >>>>>>>>>>> downstream projects may experience a tough time to adapt > > > > >>> new > > > > >>>>>>>> interfaces > > > > >>>>>>>>>>> while Flink keeps in 1.x versions so that users may > > > > >> expect > > > > >>> it > > > > >>>>> as > > > > >>>>>> an > > > > >>>>>>>>> easy > > > > >>>>>>>>>>> task. From my experience, it's really challenge to > > > > >> maintain > > > > >>>>>>>>>>> compatibility between multiple versions of Flink while > > > > >>>>>> significant > > > > >>>>>>>>>> changes > > > > >>>>>>>>>>> made but sharing 1.x version series - users may not be > > > > >>> aware > > > > >>>>> that > > > > >>>>>>>> it's > > > > >>>>>>>>>>> almost a major version bump. > > > > >>>>>>>>>>> > > > > >>>>>>>>>>> Best, > > > > >>>>>>>>>>> tison. > > > > >>>>>>>>>>> > > > > >>>>>>>>>>> > > > > >>>>>>>>>>> tison <wander4...@gmail.com> 于2022年6月6日周一 21:51写道： > > > > >>>>>>>>>>> > > > > >>>>>>>>>>>> One question from my side: > > > > >>>>>>>>>>>> > > > > >>>>>>>>>>>> As SourceFunction a @Public interface, we cannot remove > > > > >> it > > > > >>>>>> before > > > > >>>>>>>>> doing > > > > >>>>>>>>>>>> a major version bump (Flink 2.0). > > > > >>>>>>>>>>>> > > > > >>>>>>>>>>>> Of course it's not a blocker to make such deprecation > > > > >> and > > > > >>>> let > > > > >>>>>> the > > > > >>>>>>>> new > > > > >>>>>>>>>>>> interface step in. My question is whether we have a plan > > > > >>> to > > > > >>>>>>> finally > > > > >>>>>>>>>> remove > > > > >>>>>>>>>>>> the deprecated interfaces, or postpone it until a clear > > > > >>> plan > > > > >>>>> of > > > > >>>>>>>> Flink > > > > >>>>>>>>>> 2.0? > > > > >>>>>>>>>>>> > > > > >>>>>>>>>>>> Best, > > > > >>>>>>>>>>>> tison. > > > > >>>>>>>>>>>> > > > > >>>>>>>>>>>> > > > > >>>>>>>>>>>> David Anderson <dander...@apache.org> 于2022年6月6日周一 > > > > >>> 21:35写道： > > > > >>>>>>>>>>>> > > > > >>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>> David, can you elaborate why you need watermark > > > > >>>> generation > > > > >>>>> in > > > > >>>>>>> the > > > > >>>>>>>>>>>>> source > > > > >>>>>>>>>>>>>> for your data generators? > > > > >>>>>>>>>>>>> > > > > >>>>>>>>>>>>> > > > > >>>>>>>>>>>>> The training exercises should strive to provide > > > > >> examples > > > > >>> of > > > > >>>>>> best > > > > >>>>>>>>>>>>> practices. > > > > >>>>>>>>>>>>> If the exercises and their solutions use > > > > >>>>>>>>>>>>> > > > > >>>>>>>>>>>>> env.fromSource(source, > > > > >> WatermarkStrategy.noWatermarks(), > > > > >>>>>>>>>>>>> "name-of-source") > > > > >>>>>>>>>>>>> .map(...) > > > > >>>>>>>>>>>>> .assignTimestampsAndWatermarks(...) > > > > >>>>>>>>>>>>> > > > > >>>>>>>>>>>>> this will help establish this anti-pattern as the > > > > >> normal > > > > >>>> way > > > > >>>>> of > > > > >>>>>>>> doing > > > > >>>>>>>>>>>>> things. > > > > >>>>>>>>>>>>> > > > > >>>>>>>>>>>>> Most new Flink users are using a KafkaSource with a > > > > >>>>>> noWatermarks > > > > >>>>>>>>>> strategy > > > > >>>>>>>>>>>>> and a SimpleStringSchema, followed by a map that does > > > > >> the > > > > >>>>> real > > > > >>>>>>>>>>>>> deserialization, followed by the real watermarking -- > > > > >>>> because > > > > >>>>>>> they > > > > >>>>>>>>>> aren't > > > > >>>>>>>>>>>>> seeing examples that teach how these interfaces are > > > > >> meant > > > > >>>> to > > > > >>>>> be > > > > >>>>>>>> used. > > > > >>>>>>>>>>>>> > > > > >>>>>>>>>>>>> When we redo the sources used in training exercises, I > > > > >>> want > > > > >>>>> to > > > > >>>>>>>> avoid > > > > >>>>>>>>>>>>> these > > > > >>>>>>>>>>>>> pitfalls. > > > > >>>>>>>>>>>>> > > > > >>>>>>>>>>>>> David > > > > >>>>>>>>>>>>> > > > > >>>>>>>>>>>>> On Mon, Jun 6, 2022 at 9:12 AM Konstantin Knauf < > > > > >>>>>>> kna...@apache.org > > > > >>>>>>>>> > > > > >>>>>>>>>>>>> wrote: > > > > >>>>>>>>>>>>> > > > > >>>>>>>>>>>>>> Hi everyone, > > > > >>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>> very interesting thread. The proposal for deprecation > > > > >>>> seems > > > > >>>>>> to > > > > >>>>>>>> have > > > > >>>>>>>>>>>>> sparked > > > > >>>>>>>>>>>>>> a very important discussion. Do we what users > > > > >> struggle > > > > >>>> with > > > > >>>>>>>>>>>>> specifically? > > > > >>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>> Speaking for myself, when I upgrade flink-faker to > > > > >> the > > > > >>>> new > > > > >>>>>>> Source > > > > >>>>>>>>> API > > > > >>>>>>>>>>>>> an > > > > >>>>>>>>>>>>>> unbounded version of the NumberSequenceSource would > > > > >>> have > > > > >>>>> been > > > > >>>>>>>> all I > > > > >>>>>>>>>>>>> needed, > > > > >>>>>>>>>>>>>> but that's just the data generator use case. I think, > > > > >>>> that > > > > >>>>>> one > > > > >>>>>>>>> could > > > > >>>>>>>>>> be > > > > >>>>>>>>>>>>>> solved quite easily. David, can you elaborate why you > > > > >>>> need > > > > >>>>>>>>> watermark > > > > >>>>>>>>>>>>>> generation in the source for your data generators? > > > > >>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>> Cheers, > > > > >>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>> Konstantin > > > > >>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>> Am So., 5. Juni 2022 um 17:48 Uhr schrieb Piotr > > > > >>> Nowojski > > > > >>>> < > > > > >>>>>>>>>>>>>> pnowoj...@apache.org>: > > > > >>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> Also +1 to what David has written. But it doesn't > > > > >>> mean > > > > >>>> we > > > > >>>>>>>> should > > > > >>>>>>>>> be > > > > >>>>>>>>>>>>>> waiting > > > > >>>>>>>>>>>>>>> indefinitely to deprecate SourceFunction. > > > > >>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> Best, > > > > >>>>>>>>>>>>>>> Piotrek > > > > >>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> niedz., 5 cze 2022 o 16:46 Jark Wu < > > > > >> imj...@gmail.com > > > > >>>> > > > > >>>>>>>>> napisał(a): > > > > >>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>> +1 to David's point. > > > > >>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>> Usually, when we deprecate some interfaces, we > > > > >>> should > > > > >>>>>> point > > > > >>>>>>>>> users > > > > >>>>>>>>>>>>> to > > > > >>>>>>>>>>>>>> use > > > > >>>>>>>>>>>>>>>> the recommended alternatives. > > > > >>>>>>>>>>>>>>>> However, implementing the new Source interface > > > > >> for > > > > >>>> some > > > > >>>>>>>> simple > > > > >>>>>>>>>>>>>> scenarios > > > > >>>>>>>>>>>>>>> is > > > > >>>>>>>>>>>>>>>> too challenging and complex. > > > > >>>>>>>>>>>>>>>> We also found it isn't easy to push the internal > > > > >>>>>> connector > > > > >>>>>>> to > > > > >>>>>>>>>>>>> upgrade > > > > >>>>>>>>>>>>>> to > > > > >>>>>>>>>>>>>>>> the new Source because > > > > >>>>>>>>>>>>>>>> "FLIP-27 are hard to understand, while > > > > >>> SourceFunction > > > > >>>>> is > > > > >>>>>>>> easy". > > > > >>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>> +1 to make implementing a simple Source easier > > > > >>> before > > > > >>>>>>>>> deprecating > > > > >>>>>>>>>>>>>>>> SourceFunction. > > > > >>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>> Best, > > > > >>>>>>>>>>>>>>>> Jark > > > > >>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>> On Sun, 5 Jun 2022 at 07:29, Jingsong Lee < > > > > >>>>>>>>>> lzljs3620...@apache.org > > > > >>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> wrote: > > > > >>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>> +1 to David and Ingo. > > > > >>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>> Before deprecate and remove SourceFunction, we > > > > >>>> should > > > > >>>>>>> have > > > > >>>>>>>>> some > > > > >>>>>>>>>>>>>> easier > > > > >>>>>>>>>>>>>>>> APIs > > > > >>>>>>>>>>>>>>>>> to wrap new Source, the cost to write a new > > > > >>> Source > > > > >>>> is > > > > >>>>>> too > > > > >>>>>>>>> high > > > > >>>>>>>>>>>>> now. > > > > >>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>> Ingo Bürk <airbla...@apache.org>于2022年6月5日 > > > > >>>>> 周日05:32写道： > > > > >>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>> I +1 everything David said. The new Source > > > > >> API > > > > >>>>> raised > > > > >>>>>>> the > > > > >>>>>>>>>>>>>> complexity > > > > >>>>>>>>>>>>>>>>>> significantly. It's great to have such a > > > > >> rich, > > > > >>>>>> powerful > > > > >>>>>>>> API > > > > >>>>>>>>>>>>> that > > > > >>>>>>>>>>>>>> can > > > > >>>>>>>>>>>>>>> do > > > > >>>>>>>>>>>>>>>>>> everything, but in the process we lost the > > > > >>>> ability > > > > >>>>> to > > > > >>>>>>>>> onboard > > > > >>>>>>>>>>>>>> people > > > > >>>>>>>>>>>>>>> to > > > > >>>>>>>>>>>>>>>>>> the APIs. > > > > >>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>> Best > > > > >>>>>>>>>>>>>>>>>> Ingo > > > > >>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>> On 04.06.22 21:21, David Anderson wrote: > > > > >>>>>>>>>>>>>>>>>>> I'm in favor of this, but I think we need > > > > >> to > > > > >>>> make > > > > >>>>>> it > > > > >>>>>>>>> easier > > > > >>>>>>>>>>>>> to > > > > >>>>>>>>>>>>>>>>> implement > > > > >>>>>>>>>>>>>>>>>>> data generators and test sources. As things > > > > >>>> stand > > > > >>>>>> in > > > > >>>>>>>>> 1.15, > > > > >>>>>>>>>>>>> unless > > > > >>>>>>>>>>>>>>> you > > > > >>>>>>>>>>>>>>>>> can > > > > >>>>>>>>>>>>>>>>>>> be satisfied with using a > > > > >>> NumberSequenceSource > > > > >>>>>>> followed > > > > >>>>>>>>> by > > > > >>>>>>>>>> a > > > > >>>>>>>>>>>>> map, > > > > >>>>>>>>>>>>>>>>> things > > > > >>>>>>>>>>>>>>>>>>> get quite complicated. I looked into > > > > >>> reworking > > > > >>>>> the > > > > >>>>>>> data > > > > >>>>>>>>>>>>>> generators > > > > >>>>>>>>>>>>>>>> used > > > > >>>>>>>>>>>>>>>>>> in > > > > >>>>>>>>>>>>>>>>>>> the training exercises, and got discouraged > > > > >>> by > > > > >>>>> the > > > > >>>>>>>> amount > > > > >>>>>>>>>> of > > > > >>>>>>>>>>>>> work > > > > >>>>>>>>>>>>>>>>>> involved. > > > > >>>>>>>>>>>>>>>>>>> (The sources used in the training want to > > > > >> be > > > > >>>>>>> unbounded, > > > > >>>>>>>>> and > > > > >>>>>>>>>>>>> need > > > > >>>>>>>>>>>>>>>>>>> watermarking in the sources, which means > > > > >> that > > > > >>>>> using > > > > >>>>>>>>>>>>>>>>> NumberSequenceSource > > > > >>>>>>>>>>>>>>>>>>> isn't an option.) > > > > >>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>> I think the proposed deprecation will be > > > > >>> better > > > > >>>>>>>> received > > > > >>>>>>>>> if > > > > >>>>>>>>>>>>> it > > > > >>>>>>>>>>>>>> can > > > > >>>>>>>>>>>>>>> be > > > > >>>>>>>>>>>>>>>>>>> accompanied by something that makes > > > > >>>> implementing > > > > >>>>> a > > > > >>>>>>>> simple > > > > >>>>>>>>>>>>> Source > > > > >>>>>>>>>>>>>>>> easier > > > > >>>>>>>>>>>>>>>>>>> than it is now. People are continuing to > > > > >>>>> implement > > > > >>>>>>> new > > > > >>>>>>>>>>>>>>>> SourceFunctions > > > > >>>>>>>>>>>>>>>>>>> because the interfaces defined by FLIP-27 > > > > >> are > > > > >>>>> hard > > > > >>>>>> to > > > > >>>>>>>>>>>>> understand, > > > > >>>>>>>>>>>>>>>> while > > > > >>>>>>>>>>>>>>>>>>> SourceFunction is easy. Alex, I believe you > > > > >>>> were > > > > >>>>>>>> looking > > > > >>>>>>>>>> into > > > > >>>>>>>>>>>>>>>>>> implementing > > > > >>>>>>>>>>>>>>>>>>> an easier-to-use building block that could > > > > >> be > > > > >>>>> used > > > > >>>>>> in > > > > >>>>>>>>>>>>> situations > > > > >>>>>>>>>>>>>>> like > > > > >>>>>>>>>>>>>>>>>> this. > > > > >>>>>>>>>>>>>>>>>>> Can we get something like that in place > > > > >>> first? > > > > >>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>> David > > > > >>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>> On Fri, Jun 3, 2022 at 4:52 PM Jing Ge < > > > > >>>>>>>>> j...@ververica.com > > > > >>>>>>>>>>> > > > > >>>>>>>>>>>>>> wrote: > > > > >>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>> Hi, > > > > >>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>> Thanks Alex for driving this! > > > > >>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>> +1 To give the Flink developers, > > > > >> especially > > > > >>>>>>> Connector > > > > >>>>>>>>>>>>> developers > > > > >>>>>>>>>>>>>>> the > > > > >>>>>>>>>>>>>>>>>> clear > > > > >>>>>>>>>>>>>>>>>>>> signal that the new Source API is > > > > >>> recommended > > > > >>>>>>>> according > > > > >>>>>>>>> to > > > > >>>>>>>>>>>>>>> FLIP-27, > > > > >>>>>>>>>>>>>>>> we > > > > >>>>>>>>>>>>>>>>>>>> should mark them as deprecated. > > > > >>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>> There are some open questions to discuss: > > > > >>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>> 1. Do we need to mark all > > > > >>>>> subinterfaces/subclasses > > > > >>>>>>> as > > > > >>>>>>>>>>>>>> deprecated? > > > > >>>>>>>>>>>>>>>> e.g. > > > > >>>>>>>>>>>>>>>>>>>> FromElementsFunction, etc. there are many. > > > > >>>> What > > > > >>>>>> are > > > > >>>>>>>> the > > > > >>>>>>>>>>>>>>>> replacements? > > > > >>>>>>>>>>>>>>>>>>>> 2. Do we need to mark all subclasses that > > > > >>> have > > > > >>>>>>>>> replacement > > > > >>>>>>>>>>>>> as > > > > >>>>>>>>>>>>>>>>>> deprecated? > > > > >>>>>>>>>>>>>>>>>>>> e.g. ExternallyInducedSource whose > > > > >>> replacement > > > > >>>>>>> class, > > > > >>>>>>>>> if I > > > > >>>>>>>>>>>>> am > > > > >>>>>>>>>>>>>> not > > > > >>>>>>>>>>>>>>>>>> mistaken, > > > > >>>>>>>>>>>>>>>>>>>> ExternallyInducedSourceReader is > > > > >>> @Experimental > > > > >>>>>>>>>>>>>>>>>>>> 3. Do we need to mark all related test > > > > >>> utility > > > > >>>>>>> classes > > > > >>>>>>>>> as > > > > >>>>>>>>>>>>>>>> deprecated? > > > > >>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>> I think it might make sense to create an > > > > >>>>> umbrella > > > > >>>>>>>> ticket > > > > >>>>>>>>>> to > > > > >>>>>>>>>>>>>> cover > > > > >>>>>>>>>>>>>>>> all > > > > >>>>>>>>>>>>>>>>> of > > > > >>>>>>>>>>>>>>>>>>>> these with the following process: > > > > >>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>> 1. Mark SourceFunction as deprecated asap. > > > > >>>>>>>>>>>>>>>>>>>> 2. Mark subinterfaces and subclasses as > > > > >>>>>> deprecated, > > > > >>>>>>> if > > > > >>>>>>>>>>>>> there are > > > > >>>>>>>>>>>>>>>>>> graduated > > > > >>>>>>>>>>>>>>>>>>>> replacements. Good example is that > > > > >>> KafkaSource > > > > >>>>>>>> replaced > > > > >>>>>>>>>>>>>>>> KafkaConsumer > > > > >>>>>>>>>>>>>>>>>> which > > > > >>>>>>>>>>>>>>>>>>>> has been marked as deprecated. > > > > >>>>>>>>>>>>>>>>>>>> 3. Do not mark subinterfaces and > > > > >> subclasses > > > > >>> as > > > > >>>>>>>>> deprecated, > > > > >>>>>>>>>>>>> if > > > > >>>>>>>>>>>>>>>>>> replacement > > > > >>>>>>>>>>>>>>>>>>>> classes are still experimental, check if > > > > >> it > > > > >>> is > > > > >>>>>> time > > > > >>>>>>> to > > > > >>>>>>>>>>>>> graduate > > > > >>>>>>>>>>>>>>>> them. > > > > >>>>>>>>>>>>>>>>>> After > > > > >>>>>>>>>>>>>>>>>>>> graduation, go to step 2. It might take a > > > > >>>> while > > > > >>>>>> for > > > > >>>>>>>>>>>>> graduation. > > > > >>>>>>>>>>>>>>>>>>>> 4. Do not mark subinterfaces and > > > > >> subclasses > > > > >>> as > > > > >>>>>>>>> deprecated, > > > > >>>>>>>>>>>>> if > > > > >>>>>>>>>>>>>> the > > > > >>>>>>>>>>>>>>>>>>>> replacement classes are experimental and > > > > >> are > > > > >>>> too > > > > >>>>>>> young > > > > >>>>>>>>> to > > > > >>>>>>>>>>>>>>> graduate. > > > > >>>>>>>>>>>>>>>> We > > > > >>>>>>>>>>>>>>>>>> have > > > > >>>>>>>>>>>>>>>>>>>> to wait. But in this case we could create > > > > >>> new > > > > >>>>>>> tickets > > > > >>>>>>>>>> under > > > > >>>>>>>>>>>>> the > > > > >>>>>>>>>>>>>>>>> umbrella > > > > >>>>>>>>>>>>>>>>>>>> ticket. > > > > >>>>>>>>>>>>>>>>>>>> 5. Do not mark subinterfaces and > > > > >> subclasses > > > > >>> as > > > > >>>>>>>>> deprecated, > > > > >>>>>>>>>>>>> if > > > > >>>>>>>>>>>>>>> there > > > > >>>>>>>>>>>>>>>> is > > > > >>>>>>>>>>>>>>>>>> no > > > > >>>>>>>>>>>>>>>>>>>> replacement at all. We have to create new > > > > >>>>> tickets > > > > >>>>>>> and > > > > >>>>>>>>> wait > > > > >>>>>>>>>>>>> until > > > > >>>>>>>>>>>>>>> the > > > > >>>>>>>>>>>>>>>>> new > > > > >>>>>>>>>>>>>>>>>>>> implementation has been done and > > > > >> graduated. > > > > >>> It > > > > >>>>>> will > > > > >>>>>>>>> take a > > > > >>>>>>>>>>>>>> longer > > > > >>>>>>>>>>>>>>>>> time, > > > > >>>>>>>>>>>>>>>>>>>> roughly 1,5 years. > > > > >>>>>>>>>>>>>>>>>>>> 6. For test classes, we could follow the > > > > >>> same > > > > >>>>>> rule. > > > > >>>>>>>> But > > > > >>>>>>>>> I > > > > >>>>>>>>>>>>> think > > > > >>>>>>>>>>>>>>> for > > > > >>>>>>>>>>>>>>>>> some > > > > >>>>>>>>>>>>>>>>>>>> cases, we could consider doing the > > > > >>> replacement > > > > >>>>>>>> directly > > > > >>>>>>>>>>>>> without > > > > >>>>>>>>>>>>>>>> going > > > > >>>>>>>>>>>>>>>>>>>> through the deprecation phase. > > > > >>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>> When we look back on all of these, we can > > > > >>>>> realize > > > > >>>>>> it > > > > >>>>>>>> is > > > > >>>>>>>>> a > > > > >>>>>>>>>>>>> big > > > > >>>>>>>>>>>>>> epic > > > > >>>>>>>>>>>>>>>>> (even > > > > >>>>>>>>>>>>>>>>>>>> bigger than an epic). It needs someone to > > > > >>>> drive > > > > >>>>> it > > > > >>>>>>> and > > > > >>>>>>>>>> keep > > > > >>>>>>>>>>>>>> focus > > > > >>>>>>>>>>>>>>> on > > > > >>>>>>>>>>>>>>>>> it > > > > >>>>>>>>>>>>>>>>>>>> continuously with support from the > > > > >> community > > > > >>>> and > > > > >>>>>>> push > > > > >>>>>>>>> the > > > > >>>>>>>>>>>>>>>> development > > > > >>>>>>>>>>>>>>>>>>>> towards the new Source API of FLIP-27. > > > > >>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>> If we could have consensus for this, Alex > > > > >>>> and I > > > > >>>>>>> could > > > > >>>>>>>>>>>>> create > > > > >>>>>>>>>>>>>> the > > > > >>>>>>>>>>>>>>>>>> umbrella > > > > >>>>>>>>>>>>>>>>>>>> ticket to kick it off. > > > > >>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>> Best regards, > > > > >>>>>>>>>>>>>>>>>>>> Jing > > > > >>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>> On Fri, Jun 3, 2022 at 3:54 PM Alexander > > > > >>>>> Fedulov < > > > > >>>>>>>>>>>>>>>>>> alexan...@ververica.com> > > > > >>>>>>>>>>>>>>>>>>>> wrote: > > > > >>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>>> Hi everyone, > > > > >>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>>> I would like to start the discussion > > > > >> about > > > > >>>>>> marking > > > > >>>>>>>>>>>>>>>>> SourceFunction-based > > > > >>>>>>>>>>>>>>>>>>>>> interfaces as deprecated. With the > > > > >> FLIP-27 > > > > >>>> APIs > > > > >>>>>>>>> becoming > > > > >>>>>>>>>>>>> the > > > > >>>>>>>>>>>>>> new > > > > >>>>>>>>>>>>>>>>>>>> standard, > > > > >>>>>>>>>>>>>>>>>>>>> the old ones have to be eventually phased > > > > >>>> out. > > > > >>>>>>>> Although > > > > >>>>>>>>>>>>> this > > > > >>>>>>>>>>>>>>> state > > > > >>>>>>>>>>>>>>>> is > > > > >>>>>>>>>>>>>>>>>>>> well > > > > >>>>>>>>>>>>>>>>>>>>> known within the community and no new > > > > >>>>> connectors > > > > >>>>>>>> based > > > > >>>>>>>>> on > > > > >>>>>>>>>>>>> the > > > > >>>>>>>>>>>>>> old > > > > >>>>>>>>>>>>>>>>>>>>> interfaces can be accepted into the > > > > >>> project, > > > > >>>>> the > > > > >>>>>>>>>> footprint > > > > >>>>>>>>>>>>> of > > > > >>>>>>>>>>>>>>>>>>>>> SourceFunction in the user code still > > > > >> keeps > > > > >>>>>> growing > > > > >>>>>>>>>>>>> (primarily > > > > >>>>>>>>>>>>>>> for > > > > >>>>>>>>>>>>>>>>> data > > > > >>>>>>>>>>>>>>>>>>>>> generators and test utilities). I believe > > > > >>> it > > > > >>>> is > > > > >>>>>>> best > > > > >>>>>>>> to > > > > >>>>>>>>>>>>> mark > > > > >>>>>>>>>>>>>>>>>>>> SourceFunction > > > > >>>>>>>>>>>>>>>>>>>>> as deprecated as soon as possible. What > > > > >> do > > > > >>>> you > > > > >>>>>>> think? > > > > >>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>>> Best, > > > > >>>>>>>>>>>>>>>>>>>>> Alexander Fedulov > > > > >>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>> -- > > > > >>>>>>>>>>>>>> https://twitter.com/snntrable > > > > >>>>>>>>>>>>>> https://github.com/knaufk > > > > >>>>>>>>>>>>>> > > > > >>>>>>>>>>>>> > > > > >>>>>>>>>>>> > > > > >>>>>>>>>> > > > > >>>>>>>>> > > > > >>>>>>>> > > > > >>>>>>> > > > > >>>>>> > > > > >>>>> > > > > >>>> > > > > >>> > > > > >> > > > > > > > > > > > > > > > > > > -- > > https://twitter.com/snntrable > > https://github.com/knaufk > > >
