Hi Martijn, Yes, that is exactly what I tried to say implicitly in a polite manner. :))
Best regards, Jing On Tue, Jun 14, 2022 at 8:12 PM Martijn Visser <martijnvis...@apache.org> wrote: > Hi Alex and Jing, > > According to the Flink Bylaws and FLIP guide, this change would be a > "change that impacts the public interfaces of the project". I do see little > value in creating a special deprecation FLIP, but I do think it requires a > [VOTE] thread which follows the FLIP approval process. So basically I would > create a VOTE thread while linking to this discussion thread for context. > > Best regards, > > Martijn > > [1] https://cwiki.apache.org/confluence/display/FLINK/Flink+Bylaws > [2] > > https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals#FlinkImprovementProposals-Whatisconsidereda%22majorchange%22thatneedsaFLIP > ? > > Op di 14 jun. 2022 om 17:22 schreef Alexander Fedulov < > alexan...@ververica.com>: > > > Hi Jing, > > > > I don't think we do pure deprecation FLIPs. I am also OK if we consider > > consensus in this thread enough to proceed with opening the actual > > deprecation PRs for the first two items of the umbrella ticket. In that > > case, we can skip the vote. > > > > What do people prefer? Do you think we need to do a formal vote for > > this change or should we simply kick it off with the first deprecations > > based on the discussion above? > > > > Best, > > Alexander Fedulov > > > > On Tue, Jun 14, 2022 at 4:26 PM Jing Ge <j...@ververica.com> wrote: > > > > > Hi Alex, > > > > > > I guess you are meaning to start a new voting thread following the FLIP > > > discussion&voting concept. Looking forward to it! > > > > > > Best regards, > > > Jing > > > > > > On Tue, Jun 14, 2022 at 3:35 PM Alexander Fedulov < > > alexan...@ververica.com > > > > > > > wrote: > > > > > > > Hi Becket, > > > > > > > > thanks for your feedback. As proposed, I started an umbrella ticket > [1] > > > to > > > > collect all the > > > > steps needed. Please add any missing items. > > > > > > > > Judging by the discussion on this thread I propose to open a vote on > > the > > > > first two > > > > subtasks since they are immediately actionable [2] [3]. > > > > > > > > [1] https://issues.apache.org/jira/browse/FLINK-28045 > > > > [2] https://issues.apache.org/jira/browse/FLINK-28046 > > > > [3] https://issues.apache.org/jira/browse/FLINK-28047 > > > > > > > > Best, > > > > Alexander Fedulov > > > > > > > > > > > > On Tue, Jun 14, 2022 at 4:37 AM Becket Qin <becket....@gmail.com> > > wrote: > > > > > > > > > In general, I'll give a big +1 to deprecating the SourceFunction. > > > > > > > > > > That said, it is indeed worth looking into what might be missing or > > > less > > > > > easy to implement with FLIP-27 Source compared with the > > SourceFunction. > > > > > Maybe we can just compile a list of things to do in order to fully > > > > > deprecate the SourceFunction. As far as I am aware of, there are > two > > > > things > > > > > that need to be taken care of： > > > > > > > > > > 1. A simple high level API, as Jing mentioned, that makes simple > > cases > > > > that > > > > > do not involve the split enumerator easier. Ideally this should be > as > > > > > simple as SourceFunction, if not simpler. Off the top of my head, I > > > > think a > > > > > default no-op split enumerator will just do the work. And the data > > > > > generator of FLIP-238 could be an implementation using this high > > level > > > > API. > > > > > > > > > > 2. FLIP-208, which allows users to stop the job upon receiving a > > record > > > > in > > > > > the stream. > > > > > > > > > > Is there anything else that we have heard from the users / > connector > > > > > developers that needs some attention? > > > > > > > > > > Thanks, > > > > > > > > > > Jiangjie (Becket) Qin > > > > > > > > > > > > > > > > > > > > On Fri, Jun 10, 2022 at 3:25 PM David Anderson < > dander...@apache.org > > > > > > > > wrote: > > > > > > > > > > > +1 for deprecating SourceFunction from me as well. And a big > THANK > > > YOU > > > > to > > > > > > Alex Fedulov for bringing forward FLIP-238. > > > > > > > > > > > > David > > > > > > > > > > > > On Fri, Jun 10, 2022 at 4:03 AM Lijie Wang < > > wangdachui9...@gmail.com > > > > > > > > > > wrote: > > > > > > > > > > > > > Hi all, > > > > > > > > > > > > > > Sorry for my mistake. The > `StreamExecutionEnvironment#readFiles` > > > and > > > > > can > > > > > > be > > > > > > > easily replaced by > > > > > `FileSource#forRecordStreamFormat/forBulkFileFormat`. > > > > > > I > > > > > > > have no other concerns. > > > > > > > > > > > > > > +1 to deprecate SourceFunction and deprecate the methods (in > > > > > > > StreamExecutionEnvironment) based on SourceFunction . > > > > > > > > > > > > > > Best, > > > > > > > Lijie > > > > > > > > > > > > > > Konstantin Knauf <kna...@apache.org> 于2022年6月10日周五 05:11写道： > > > > > > > > > > > > > > > Hi everyone, > > > > > > > > > > > > > > > > thank you Jing for redirecting the discussion back to the > topic > > > at > > > > > > hand. > > > > > > > I > > > > > > > > agree with all of your points. > > > > > > > > > > > > > > > > +1 to deprecate SourceFunction > > > > > > > > > > > > > > > > Is there really no replacement for the > > > > > > > StreamExecutionEnvironment#readXXX. > > > > > > > > There is already a FLIP-27 based FileSource, right? What's > > > missing > > > > to > > > > > > > > recommend using that as opposed to the the readXXX methods? > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > > > > > > > > Konstantin > > > > > > > > > > > > > > > > Am Do., 9. Juni 2022 um 20:11 Uhr schrieb Alexander Fedulov < > > > > > > > > alexan...@ververica.com>: > > > > > > > > > > > > > > > > > Hi all, > > > > > > > > > > > > > > > > > > It seems that there is some understandable cautiousness > with > > > > regard > > > > > > to > > > > > > > > > deprecating methods and subclasses that do not have > > > alternatives > > > > > just > > > > > > > > yet. > > > > > > > > > > > > > > > > > > We should probably first agree if it is in general OK for > > Flink > > > > to > > > > > > use > > > > > > > > > @Deprecated > > > > > > > > > annotation for parts of the code that do not have > > alternatives. > > > > In > > > > > > that > > > > > > > > > case, > > > > > > > > > we could add a comment along the lines of: > > > > > > > > > "This implementation is based on a deprecated > SourceFunction > > > API > > > > > that > > > > > > > > > will gradually be phased out from Flink. No direct > substitute > > > > > exists > > > > > > at > > > > > > > > the > > > > > > > > > moment. > > > > > > > > > If you want to have a more future-proof solution, consider > > > > helping > > > > > > the > > > > > > > > > project by > > > > > > > > > contributing an implementation based on the new Source > API." > > > > > > > > > > > > > > > > > > This should clearly communicate the message that usage of > > these > > > > > > > > > methods/classes > > > > > > > > > is discouraged and at the same time promote contributions > for > > > > > > > addressing > > > > > > > > > the gap. > > > > > > > > > What do you think? > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > Alexander Fedulov > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Jun 9, 2022 at 6:27 PM Ingo Bürk < > > airbla...@apache.org > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > > > > > these APIs don't expose the underlying source directly, > so > > I > > > > > don't > > > > > > > > think > > > > > > > > > > we need to worry about deprecating them as well. There's > > also > > > > > > nothing > > > > > > > > > > inherently wrong with using a deprecated API internally, > > > though > > > > > > even > > > > > > > > > > just for the experience of using our own new APIs I would > > > > > > personally > > > > > > > > say > > > > > > > > > > that they should be migrated to the new Source API. It's > > hard > > > > to > > > > > > > reason > > > > > > > > > > that users must migrate to a new API if we don't do it > > > > internally > > > > > > as > > > > > > > > > well. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Best > > > > > > > > > > Ingo > > > > > > > > > > > > > > > > > > > > On 09.06.22 15:41, Lijie Wang wrote: > > > > > > > > > > > Hi Martijn, > > > > > > > > > > > > > > > > > > > > > > I don't mean it's a blocker. Just a information. And > I'm > > > also > > > > > +1 > > > > > > > for > > > > > > > > > > this. > > > > > > > > > > > > > > > > > > > > > > Put it another way: should we migrate the > > `#readFile(...)` > > > to > > > > > new > > > > > > > API > > > > > > > > > or > > > > > > > > > > > provide a similar method "readxxx“ based on the new > > Source > > > > API? > > > > > > > > > > > > > > > > > > > > > > And if we don't migrate it, does it mean that the > > > > > > `#readFile(...)` > > > > > > > > > should > > > > > > > > > > > also be marked as deprecated? > > > > > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > > > Lijie > > > > > > > > > > > > > > > > > > > > > > Martijn Visser <martijnvis...@apache.org> 于2022年6月9日周四 > > > > > 21:03写道： > > > > > > > > > > > > > > > > > > > > > >> Hi Lijie, > > > > > > > > > > >> > > > > > > > > > > >> I don't see any problem with deprecating those methods > > at > > > > this > > > > > > > > moment, > > > > > > > > > > as > > > > > > > > > > >> long as we don't remove them until the replacements > are > > > > > > available. > > > > > > > > > > Besides > > > > > > > > > > >> that, are we sure there are no replacements already, > > > > > especially > > > > > > > with > > > > > > > > > the > > > > > > > > > > >> new FileSource? > > > > > > > > > > >> > > > > > > > > > > >> Best regards, > > > > > > > > > > >> > > > > > > > > > > >> Martijn > > > > > > > > > > >> > > > > > > > > > > >> Op do 9 jun. 2022 om 14:23 schreef Lijie Wang < > > > > > > > > > wangdachui9...@gmail.com > > > > > > > > > > >: > > > > > > > > > > >> > > > > > > > > > > >>> Hi all, > > > > > > > > > > >>> > > > > > > > > > > >>> FYI, currently, some commonly used methods in > > > > > > > > > > StreamExecutionEnvironment > > > > > > > > > > >>> are still based on the old SourceFunction (and there > is > > > no > > > > > > > > > > alternative): > > > > > > > > > > >>> `StreamExecutionEnvironment#readFile(...)` > > > > > > > > > > >>> `StreamExecutionEnvironment#readTextFile(...)` > > > > > > > > > > >>> > > > > > > > > > > >>> I think these should be migrated to the new source > API > > > > before > > > > > > > > > deprecate > > > > > > > > > > >> the > > > > > > > > > > >>> SourceFunction. > > > > > > > > > > >>> > > > > > > > > > > >>> Best, > > > > > > > > > > >>> Lijie > > > > > > > > > > >>> > > > > > > > > > > >>> Martijn Visser <martijnvis...@apache.org> > 于2022年6月9日周四 > > > > > > 16:05写道： > > > > > > > > > > >>> > > > > > > > > > > >>>> Hi all, > > > > > > > > > > >>>> > > > > > > > > > > >>>> I think implicitly we've already considered the > > > > > SourceFunction > > > > > > > and > > > > > > > > > > >>>> SinkFunction as deprecated. They are even marked as > so > > > on > > > > > the > > > > > > > > Flink > > > > > > > > > > >>> roadmap > > > > > > > > > > >>>> [1]. That also shows that connectors that are using > > > these > > > > > > > > interfaces > > > > > > > > > > >> are > > > > > > > > > > >>>> either approaching end-of-life. The fact that we're > > > > actively > > > > > > > > > migrating > > > > > > > > > > >>>> connectors from Source/SinkFunction to > > FLIP-27/FLIP-143 > > > > > (plus > > > > > > > > add-on > > > > > > > > > > >>> FLIPs) > > > > > > > > > > >>>> shows that we've already determined that target. > > > > > > > > > > >>>> > > > > > > > > > > >>>> With regards to the motivation of FLIP-27, I think > > > reading > > > > > up > > > > > > on > > > > > > > > the > > > > > > > > > > >>>> original discussion thread is also worthwhile [2] to > > see > > > > > more > > > > > > > > > context. > > > > > > > > > > >>>> FLIP-27 was also very important as it brought a > > unified > > > > > > > connector > > > > > > > > > > which > > > > > > > > > > >>> can > > > > > > > > > > >>>> support both streaming and batch (with batch being > > > > > considered > > > > > > a > > > > > > > > > > special > > > > > > > > > > >>>> case of streaming in Flink's vision). > > > > > > > > > > >>>> > > > > > > > > > > >>>> So +1 to deprecate SourceFunction. I would also > argue > > > that > > > > > we > > > > > > > > should > > > > > > > > > > >>>> already mark the SinkFunction as deprecated to avoid > > > > having > > > > > > this > > > > > > > > > > >>> discussion > > > > > > > > > > >>>> again in a couple of months. > > > > > > > > > > >>>> > > > > > > > > > > >>>> Best regards, > > > > > > > > > > >>>> > > > > > > > > > > >>>> Martijn > > > > > > > > > > >>>> > > > > > > > > > > >>>> [1] https://flink.apache.org/roadmap.html > > > > > > > > > > >>>> [2] > > > > > > > > > > > > https://lists.apache.org/thread/334co89dbhc8qpr9nvmz8t1gp4sz2c8y > > > > > > > > > > >>>> > > > > > > > > > > >>>> Op do 9 jun. 2022 om 09:48 schreef Jing Ge < > > > > > > j...@ververica.com > > > > > > > >: > > > > > > > > > > >>>> > > > > > > > > > > >>>>> Hi, > > > > > > > > > > >>>>> > > > > > > > > > > >>>>> I am very happy to see opinions from different > > > > > perspectives. > > > > > > > That > > > > > > > > > > >> will > > > > > > > > > > >>>> help > > > > > > > > > > >>>>> us understand the problem better. Thanks all for > the > > > > > > > informative > > > > > > > > > > >>>>> discussion. > > > > > > > > > > >>>>> > > > > > > > > > > >>>>> Let's see the big picture and check following facts > > > > > together: > > > > > > > > > > >>>>> > > > > > > > > > > >>>>> 1. FLIP-27 was intended to solve some technical > > issues > > > > that > > > > > > are > > > > > > > > > very > > > > > > > > > > >>>>> difficult to solve with SourceFunction[1]. When we > > say > > > > > > > > > > >> "SourceFunction > > > > > > > > > > >>> is > > > > > > > > > > >>>>> easy", well, it depends. If we take a look at the > > > > > > > implementation > > > > > > > > of > > > > > > > > > > >> the > > > > > > > > > > >>>>> Kafka connector, we will know how complicated it is > > to > > > > > build > > > > > > a > > > > > > > > > > >> serious > > > > > > > > > > >>>>> connector for production with the old > SourceFunction. > > > To > > > > > > every > > > > > > > > > > >> problem > > > > > > > > > > >>>>> there is a solution and to every solution there is > a > > > > > problem. > > > > > > > The > > > > > > > > > > >> fact > > > > > > > > > > >>> is > > > > > > > > > > >>>>> that there is no perfect but a feasible solution. > If > > we > > > > try > > > > > > to > > > > > > > > > solve > > > > > > > > > > >>>>> complicated problems, we have to expose some > > > complexity. > > > > > > > > Comparing > > > > > > > > > to > > > > > > > > > > >>>>> connectors for POC, demo, training(no offense), I > > would > > > > > also > > > > > > > > solve > > > > > > > > > > >>> issues > > > > > > > > > > >>>>> for connectors like Kafka connector that are widely > > > used > > > > in > > > > > > > > > > >> production > > > > > > > > > > >>>> with > > > > > > > > > > >>>>> higher priority. I think that should be one reason > > why > > > > > > FLIP-27 > > > > > > > > has > > > > > > > > > > >> been > > > > > > > > > > >>>>> designed and why the new source API went public. > > > > > > > > > > >>>>> > > > > > > > > > > >>>>> 2. FLIP-27 and the implementation was introduced > > > roughly > > > > at > > > > > > the > > > > > > > > end > > > > > > > > > > >> of > > > > > > > > > > >>>> 2019 > > > > > > > > > > >>>>> and went public on 19.04.2021, which means Flink > has > > > > > provided > > > > > > > two > > > > > > > > > > >>>> different > > > > > > > > > > >>>>> public/graduated source solutions for more than one > > > year. > > > > > On > > > > > > > the > > > > > > > > > day > > > > > > > > > > >>> that > > > > > > > > > > >>>>> the new source API went public, there should be a > > > > consensus > > > > > > in > > > > > > > > the > > > > > > > > > > >>>>> community that we should start the migration. Old > > > > > > > SourceFunction > > > > > > > > > > >>>> interface, > > > > > > > > > > >>>>> in the ideal case, should have been deprecated on > > that > > > > day, > > > > > > > > > otherwise > > > > > > > > > > >>> we > > > > > > > > > > >>>>> should not graduate the new source API to avoid > > > confusing > > > > > > > > > (connector) > > > > > > > > > > >>>>> developers[2]. > > > > > > > > > > >>>>> > > > > > > > > > > >>>>> 3. It is true that the new source API is hard to > > > > understand > > > > > > and > > > > > > > > > even > > > > > > > > > > >>> hard > > > > > > > > > > >>>>> to implement for simple cases. Thanks for the > > feedback. > > > > > That > > > > > > is > > > > > > > > > > >>> something > > > > > > > > > > >>>>> we need to improve. The current > design&implementation > > > > could > > > > > > be > > > > > > > > > > >>> considered > > > > > > > > > > >>>>> as the low level API. The next step is to create > the > > > high > > > > > > level > > > > > > > > API > > > > > > > > > > >> to > > > > > > > > > > >>>>> reduce some unnecessary complexity for those simple > > > > cases. > > > > > > But, > > > > > > > > > IMHO, > > > > > > > > > > >>>> this > > > > > > > > > > >>>>> should not be the prerequisite to postpone the > > > > deprecation > > > > > of > > > > > > > the > > > > > > > > > old > > > > > > > > > > >>>>> SourceFunction APIs. > > > > > > > > > > >>>>> > > > > > > > > > > >>>>> 4. As long as the old SourceFunction is not marked > as > > > > > > > deprecated, > > > > > > > > > > >>>>> developers will continue asking which one should be > > > used. > > > > > > Let's > > > > > > > > > make > > > > > > > > > > >> a > > > > > > > > > > >>>>> concrete example. If a new connector is developed > now > > > and > > > > > the > > > > > > > > > > >> developer > > > > > > > > > > >>>>> asks for a suggestion of the choice between the old > > and > > > > new > > > > > > > > source > > > > > > > > > > >> API > > > > > > > > > > >>> on > > > > > > > > > > >>>>> the ML, which one should we suggest? I think it > > should > > > be > > > > > the > > > > > > > new > > > > > > > > > > >>> Source > > > > > > > > > > >>>>> API. If a fresh new connector has been developed > with > > > the > > > > > old > > > > > > > > > > >>>>> SourceFunction API before asking for the consensus > in > > > the > > > > > > > > community > > > > > > > > > > >> and > > > > > > > > > > >>>> the > > > > > > > > > > >>>>> developer wants to merge it to the master. Should > we > > > > allow > > > > > > it? > > > > > > > If > > > > > > > > > the > > > > > > > > > > >>>>> answer of all these questions is pointing to the > new > > > > Source > > > > > > > API, > > > > > > > > > the > > > > > > > > > > >>> old > > > > > > > > > > >>>>> SourceFunction is de facto already deprecated, just > > has > > > > not > > > > > > > been > > > > > > > > > > >> marked > > > > > > > > > > >>>> as > > > > > > > > > > >>>>> @deprecated, which confuses developers even more. > > > > > > > > > > >>>>> > > > > > > > > > > >>>>> Best regards, > > > > > > > > > > >>>>> Jing > > > > > > > > > > >>>>> > > > > > > > > > > >>>>> [1] > > > > > > > > > > >>>>> > > > > > > > > > > >>>>> > > > > > > > > > > >>>> > > > > > > > > > > >>> > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-27%3A+Refactor+Source+Interface > > > > > > > > > > >>>>> [2] > > > > > > > > > > > > https://lists.apache.org/thread/7okp4y46n3o3rx5mn0t3qobrof8zxwqs > > > > > > > > > > >>>>> > > > > > > > > > > >>>>> On Wed, Jun 8, 2022 at 2:21 AM Alexander Fedulov < > > > > > > > > > > >>>> alexan...@ververica.com> > > > > > > > > > > >>>>> wrote: > > > > > > > > > > >>>>> > > > > > > > > > > >>>>>> Hey Austin, > > > > > > > > > > >>>>>> > > > > > > > > > > >>>>>> Since we are getting deeper into the > implementation > > > > > details > > > > > > of > > > > > > > > the > > > > > > > > > > >>>>>> DataGeneratorSource > > > > > > > > > > >>>>>> and it is not the main topic of this thread, I > > propose > > > > to > > > > > > move > > > > > > > > our > > > > > > > > > > >>>>>> discussion to where it belongs: [DISCUSS] FLIP-238 > > > [1]. > > > > > > Could > > > > > > > > you > > > > > > > > > > >>>> please > > > > > > > > > > >>>>>> briefly formulate your requirements to make it > > easier > > > > for > > > > > > the > > > > > > > > > > >> others > > > > > > > > > > >>> to > > > > > > > > > > >>>>>> follow? I am happy to continue this conversation > > > there. > > > > > > > > > > >>>>>> > > > > > > > > > > >>>>>> [1] > > > > > > > > > > >> > > > > > > https://lists.apache.org/thread/7gjxto1rmkpff4kl54j8nlg5db2rqhkt > > > > > > > > > > >>>>>> > > > > > > > > > > >>>>>> Best, > > > > > > > > > > >>>>>> Alexander Fedulov > > > > > > > > > > >>>>>> > > > > > > > > > > >>>>>> On Tue, Jun 7, 2022 at 6:14 PM Austin > > Cawley-Edwards < > > > > > > > > > > >>>>>> austin.caw...@gmail.com> wrote: > > > > > > > > > > >>>>>> > > > > > > > > > > >>>>>>>> @Austin, in the FLIP I mentioned above [1], the > > user > > > > is > > > > > > > > > > >> expected > > > > > > > > > > >>> to > > > > > > > > > > >>>>>>> pass a MapFunction<Long, > > > > > > > > > > >>>>>>> OUT> > > > > > > > > > > >>>>>>> to the generator. I wonder if you could have your > > > > > external > > > > > > > > client > > > > > > > > > > >>> and > > > > > > > > > > >>>>>>> polling logic wrapped in a custom > > > > > > > > > > >>>>>>> MapFunction implementation class? Would that > answer > > > > your > > > > > > > needs > > > > > > > > or > > > > > > > > > > >>> do > > > > > > > > > > >>>>> you > > > > > > > > > > >>>>>>> have some > > > > > > > > > > >>>>>>> more sophisticated scenario in mind? > > > > > > > > > > >>>>>>> > > > > > > > > > > >>>>>>> At first glance, the FLIP looks good but for this > > > case > > > > in > > > > > > > > regards > > > > > > > > > > >>> to > > > > > > > > > > >>>>> the > > > > > > > > > > >>>>>>> map function, but leaves out 1) ability to > control > > > > > polling > > > > > > > > > > >>> intervals > > > > > > > > > > >>>>> and > > > > > > > > > > >>>>>> 2) > > > > > > > > > > >>>>>>> ability to produce an unknown number of records, > > both > > > > > > > per-poll > > > > > > > > > > >> and > > > > > > > > > > >>>>>> overall > > > > > > > > > > >>>>>>> boundedness. Do you think something like this > could > > > be > > > > > > built > > > > > > > > from > > > > > > > > > > >>> the > > > > > > > > > > >>>>>> same > > > > > > > > > > >>>>>>> pieces? > > > > > > > > > > >>>>>>> I'm also wondering what handles threading, is > that > > on > > > > the > > > > > > > user > > > > > > > > or > > > > > > > > > > >>> is > > > > > > > > > > >>>>> that > > > > > > > > > > >>>>>>> part of the DataGeneratorSource? > > > > > > > > > > >>>>>>> > > > > > > > > > > >>>>>>> Best, > > > > > > > > > > >>>>>>> Austin > > > > > > > > > > >>>>>>> > > > > > > > > > > >>>>>>> On Tue, Jun 7, 2022 at 9:34 AM Alexander Fedulov > < > > > > > > > > > > >>>>>> alexan...@ververica.com> > > > > > > > > > > >>>>>>> wrote: > > > > > > > > > > >>>>>>> > > > > > > > > > > >>>>>>>> Hi everyone, > > > > > > > > > > >>>>>>>> > > > > > > > > > > >>>>>>>> Thanks for all the input and a lively > discussion. > > It > > > > > seems > > > > > > > > that > > > > > > > > > > >>>> there > > > > > > > > > > >>>>>> is > > > > > > > > > > >>>>>>> a > > > > > > > > > > >>>>>>>> consensus that due to > > > > > > > > > > >>>>>>>> the inherent complexity of FLIP-27 sources we > > should > > > > > > provide > > > > > > > > > > >> more > > > > > > > > > > >>>>>>>> user-facing utilities to bridge > > > > > > > > > > >>>>>>>> the gap between the existing > SourceFunction-based > > > > > > > > functionality > > > > > > > > > > >>> and > > > > > > > > > > >>>>> the > > > > > > > > > > >>>>>>> new > > > > > > > > > > >>>>>>>> APIs. > > > > > > > > > > >>>>>>>> > > > > > > > > > > >>>>>>>> To start addressing this I picked the issue that > > > David > > > > > > > raised > > > > > > > > > > >> and > > > > > > > > > > >>>>> many > > > > > > > > > > >>>>>>>> upvoted. Here is a proposal > > > > > > > > > > >>>>>>>> for the new DataGeneratorSource: FLIP-238 [1]. > > > Please > > > > > > take > > > > > > > a > > > > > > > > > > >>>> look, I > > > > > > > > > > >>>>>> am > > > > > > > > > > >>>>>>>> going to open a separate > > > > > > > > > > >>>>>>>> discussion thread on it shortly. > > > > > > > > > > >>>>>>>> > > > > > > > > > > >>>>>>>> Jing also raised some great points regarding the > > > > > > interfaces > > > > > > > > and > > > > > > > > > > >>>>>>> subclasses. > > > > > > > > > > >>>>>>>> It seems to me that > > > > > > > > > > >>>>>>>> what might actually help is some sort of a "soft > > > > > > > deprecation" > > > > > > > > > > >>>> concept > > > > > > > > > > >>>>>> and > > > > > > > > > > >>>>>>>> annotation. It could be > > > > > > > > > > >>>>>>>> used in places where we do not have an > alternative > > > > > > > > > > >> implementation > > > > > > > > > > >>>>> yet, > > > > > > > > > > >>>>>>> but > > > > > > > > > > >>>>>>>> we clearly want > > > > > > > > > > >>>>>>>> to indicate that continuing to build on top of > > these > > > > > > > > interfaces > > > > > > > > > > >>> is > > > > > > > > > > >>>>>>>> discouraged. The area of > > > > > > > > > > >>>>>>>> impact of deprecating all SourceFunction > > subclasses > > > is > > > > > > > rather > > > > > > > > > > >>> big, > > > > > > > > > > >>>>> and > > > > > > > > > > >>>>>> we > > > > > > > > > > >>>>>>>> can expect it to > > > > > > > > > > >>>>>>>> take a while. The hope would be that if in the > > > > meantime > > > > > > > > someone > > > > > > > > > > >>>> finds > > > > > > > > > > >>>>>>>> themselves using one of > > > > > > > > > > >>>>>>>> such old APIs, the "soft deprecation" annotation > > > will > > > > > be a > > > > > > > > > > >> clear > > > > > > > > > > >>>>>>> indication > > > > > > > > > > >>>>>>>> and encouragement to > > > > > > > > > > >>>>>>>> work on introducing an alternative FLIP-27-based > > > > > > > > implementation > > > > > > > > > > >>>>>> instead. > > > > > > > > > > >>>>>>>> > > > > > > > > > > >>>>>>>> @Austin, in the FLIP I mentioned above [1], the > > user > > > > is > > > > > > > > > > >> expected > > > > > > > > > > >>> to > > > > > > > > > > >>>>>>>> pass a MapFunction<Long, > > > > > > > > > > >>>>>>>> OUT> > > > > > > > > > > >>>>>>>> to the generator. I wonder if you could have > your > > > > > external > > > > > > > > > > >> client > > > > > > > > > > >>>> and > > > > > > > > > > >>>>>>>> polling logic wrapped in a custom > > > > > > > > > > >>>>>>>> MapFunction implementation class? Would that > > answer > > > > your > > > > > > > needs > > > > > > > > > > >> or > > > > > > > > > > >>>> do > > > > > > > > > > >>>>>> you > > > > > > > > > > >>>>>>>> have some > > > > > > > > > > >>>>>>>> more sophisticated scenario in mind? > > > > > > > > > > >>>>>>>> > > > > > > > > > > >>>>>>>> [1] https://cwiki.apache.org/confluence/x/9Av1D > > > > > > > > > > >>>>>>>> Best, > > > > > > > > > > >>>>>>>> Alexander Fedulov > > > > > > > > > > >>>>>>>> > > > > > > > > > > >>>>>>>> On Mon, Jun 6, 2022 at 7:08 PM Austin > > > Cawley-Edwards < > > > > > > > > > > >>>>>>>> austin.caw...@gmail.com> wrote: > > > > > > > > > > >>>>>>>> > > > > > > > > > > >>>>>>>>> Thanks for the nice discussion all. > > > > > > > > > > >>>>>>>>> > > > > > > > > > > >>>>>>>>> I was recently trying to implement a very > simple > > > > > polling > > > > > > > > > > >> source > > > > > > > > > > >>>> and > > > > > > > > > > >>>>>>>>> would've loved a higher-level base to work > from. > > > I'm > > > > > > > > > > >> wondering > > > > > > > > > > >>> if > > > > > > > > > > >>>>> in > > > > > > > > > > >>>>>>>>> addition to the data generator use cases, it > > would > > > be > > > > > > good > > > > > > > to > > > > > > > > > > >>>>>> support a > > > > > > > > > > >>>>>>>>> simple non-parallel polling abstraction to make > > it > > > > > easier > > > > > > > to, > > > > > > > > > > >>> for > > > > > > > > > > >>>>>>>> instance, > > > > > > > > > > >>>>>>>>> start prototyping with data in existing APIs > > > without > > > > > > > adding a > > > > > > > > > > >>>> Kafka > > > > > > > > > > >>>>>> or > > > > > > > > > > >>>>>>>> such > > > > > > > > > > >>>>>>>>> in the middle. > > > > > > > > > > >>>>>>>>> > > > > > > > > > > >>>>>>>>> Best, > > > > > > > > > > >>>>>>>>> Austin > > > > > > > > > > >>>>>>>>> > > > > > > > > > > >>>>>>>>> On Mon, Jun 6, 2022 at 10:02 AM tison < > > > > > > > wander4...@gmail.com> > > > > > > > > > > >>>>> wrote: > > > > > > > > > > >>>>>>>>> > > > > > > > > > > >>>>>>>>>> Well. It's a bit off-topic. For deprecating > > > > > > SourceFunction > > > > > > > > > > >> as > > > > > > > > > > >>>>>> FLIP-27 > > > > > > > > > > >>>>>>>>>> series works go ahead, +1 from my side. It's a > > > > > > significant > > > > > > > > > > >>> work > > > > > > > > > > >>>>>>> towards > > > > > > > > > > >>>>>>>>> the > > > > > > > > > > >>>>>>>>>> unification of batch and streaming effort :) > > > > > > > > > > >>>>>>>>>> > > > > > > > > > > >>>>>>>>>> Best, > > > > > > > > > > >>>>>>>>>> tison. > > > > > > > > > > >>>>>>>>>> > > > > > > > > > > >>>>>>>>>> > > > > > > > > > > >>>>>>>>>> tison <wander4...@gmail.com> 于2022年6月6日周一 > > > 21:54写道： > > > > > > > > > > >>>>>>>>>> > > > > > > > > > > >>>>>>>>>>> The starting point of the version bump and > > > removal > > > > > > > > > > >> question > > > > > > > > > > >>>> is > > > > > > > > > > >>>>>> that > > > > > > > > > > >>>>>>>>>>> downstream projects may experience a tough > time > > > to > > > > > > adapt > > > > > > > > > > >>> new > > > > > > > > > > >>>>>>>> interfaces > > > > > > > > > > >>>>>>>>>>> while Flink keeps in 1.x versions so that > users > > > may > > > > > > > > > > >> expect > > > > > > > > > > >>> it > > > > > > > > > > >>>>> as > > > > > > > > > > >>>>>> an > > > > > > > > > > >>>>>>>>> easy > > > > > > > > > > >>>>>>>>>>> task. From my experience, it's really > challenge > > > to > > > > > > > > > > >> maintain > > > > > > > > > > >>>>>>>>>>> compatibility between multiple versions of > > Flink > > > > > while > > > > > > > > > > >>>>>> significant > > > > > > > > > > >>>>>>>>>> changes > > > > > > > > > > >>>>>>>>>>> made but sharing 1.x version series - users > may > > > not > > > > > be > > > > > > > > > > >>> aware > > > > > > > > > > >>>>> that > > > > > > > > > > >>>>>>>> it's > > > > > > > > > > >>>>>>>>>>> almost a major version bump. > > > > > > > > > > >>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>> Best, > > > > > > > > > > >>>>>>>>>>> tison. > > > > > > > > > > >>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>> tison <wander4...@gmail.com> 于2022年6月6日周一 > > > 21:51写道： > > > > > > > > > > >>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>> One question from my side: > > > > > > > > > > >>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>> As SourceFunction a @Public interface, we > > cannot > > > > > > remove > > > > > > > > > > >> it > > > > > > > > > > >>>>>> before > > > > > > > > > > >>>>>>>>> doing > > > > > > > > > > >>>>>>>>>>>> a major version bump (Flink 2.0). > > > > > > > > > > >>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>> Of course it's not a blocker to make such > > > > > deprecation > > > > > > > > > > >> and > > > > > > > > > > >>>> let > > > > > > > > > > >>>>>> the > > > > > > > > > > >>>>>>>> new > > > > > > > > > > >>>>>>>>>>>> interface step in. My question is whether we > > > have > > > > a > > > > > > plan > > > > > > > > > > >>> to > > > > > > > > > > >>>>>>> finally > > > > > > > > > > >>>>>>>>>> remove > > > > > > > > > > >>>>>>>>>>>> the deprecated interfaces, or postpone it > > until > > > a > > > > > > clear > > > > > > > > > > >>> plan > > > > > > > > > > >>>>> of > > > > > > > > > > >>>>>>>> Flink > > > > > > > > > > >>>>>>>>>> 2.0? > > > > > > > > > > >>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>> Best, > > > > > > > > > > >>>>>>>>>>>> tison. > > > > > > > > > > >>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>> David Anderson <dander...@apache.org> > > > > 于2022年6月6日周一 > > > > > > > > > > >>> 21:35写道： > > > > > > > > > > >>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>>>> David, can you elaborate why you need > > > watermark > > > > > > > > > > >>>> generation > > > > > > > > > > >>>>> in > > > > > > > > > > >>>>>>> the > > > > > > > > > > >>>>>>>>>>>>> source > > > > > > > > > > >>>>>>>>>>>>>> for your data generators? > > > > > > > > > > >>>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>>> The training exercises should strive to > > provide > > > > > > > > > > >> examples > > > > > > > > > > >>> of > > > > > > > > > > >>>>>> best > > > > > > > > > > >>>>>>>>>>>>> practices. > > > > > > > > > > >>>>>>>>>>>>> If the exercises and their solutions use > > > > > > > > > > >>>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>>> env.fromSource(source, > > > > > > > > > > >> WatermarkStrategy.noWatermarks(), > > > > > > > > > > >>>>>>>>>>>>> "name-of-source") > > > > > > > > > > >>>>>>>>>>>>> .map(...) > > > > > > > > > > >>>>>>>>>>>>> .assignTimestampsAndWatermarks(...) > > > > > > > > > > >>>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>>> this will help establish this anti-pattern > as > > > the > > > > > > > > > > >> normal > > > > > > > > > > >>>> way > > > > > > > > > > >>>>> of > > > > > > > > > > >>>>>>>> doing > > > > > > > > > > >>>>>>>>>>>>> things. > > > > > > > > > > >>>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>>> Most new Flink users are using a > KafkaSource > > > > with a > > > > > > > > > > >>>>>> noWatermarks > > > > > > > > > > >>>>>>>>>> strategy > > > > > > > > > > >>>>>>>>>>>>> and a SimpleStringSchema, followed by a map > > > that > > > > > does > > > > > > > > > > >> the > > > > > > > > > > >>>>> real > > > > > > > > > > >>>>>>>>>>>>> deserialization, followed by the real > > > > watermarking > > > > > -- > > > > > > > > > > >>>> because > > > > > > > > > > >>>>>>> they > > > > > > > > > > >>>>>>>>>> aren't > > > > > > > > > > >>>>>>>>>>>>> seeing examples that teach how these > > interfaces > > > > are > > > > > > > > > > >> meant > > > > > > > > > > >>>> to > > > > > > > > > > >>>>> be > > > > > > > > > > >>>>>>>> used. > > > > > > > > > > >>>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>>> When we redo the sources used in training > > > > > exercises, > > > > > > I > > > > > > > > > > >>> want > > > > > > > > > > >>>>> to > > > > > > > > > > >>>>>>>> avoid > > > > > > > > > > >>>>>>>>>>>>> these > > > > > > > > > > >>>>>>>>>>>>> pitfalls. > > > > > > > > > > >>>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>>> David > > > > > > > > > > >>>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>>> On Mon, Jun 6, 2022 at 9:12 AM Konstantin > > > Knauf < > > > > > > > > > > >>>>>>> kna...@apache.org > > > > > > > > > > >>>>>>>>> > > > > > > > > > > >>>>>>>>>>>>> wrote: > > > > > > > > > > >>>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>>>> Hi everyone, > > > > > > > > > > >>>>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>>>> very interesting thread. The proposal for > > > > > > deprecation > > > > > > > > > > >>>> seems > > > > > > > > > > >>>>>> to > > > > > > > > > > >>>>>>>> have > > > > > > > > > > >>>>>>>>>>>>> sparked > > > > > > > > > > >>>>>>>>>>>>>> a very important discussion. Do we what > > users > > > > > > > > > > >> struggle > > > > > > > > > > >>>> with > > > > > > > > > > >>>>>>>>>>>>> specifically? > > > > > > > > > > >>>>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>>>> Speaking for myself, when I upgrade > > > flink-faker > > > > to > > > > > > > > > > >> the > > > > > > > > > > >>>> new > > > > > > > > > > >>>>>>> Source > > > > > > > > > > >>>>>>>>> API > > > > > > > > > > >>>>>>>>>>>>> an > > > > > > > > > > >>>>>>>>>>>>>> unbounded version of the > > NumberSequenceSource > > > > > would > > > > > > > > > > >>> have > > > > > > > > > > >>>>> been > > > > > > > > > > >>>>>>>> all I > > > > > > > > > > >>>>>>>>>>>>> needed, > > > > > > > > > > >>>>>>>>>>>>>> but that's just the data generator use > > case. I > > > > > > think, > > > > > > > > > > >>>> that > > > > > > > > > > >>>>>> one > > > > > > > > > > >>>>>>>>> could > > > > > > > > > > >>>>>>>>>> be > > > > > > > > > > >>>>>>>>>>>>>> solved quite easily. David, can you > > elaborate > > > > why > > > > > > you > > > > > > > > > > >>>> need > > > > > > > > > > >>>>>>>>> watermark > > > > > > > > > > >>>>>>>>>>>>>> generation in the source for your data > > > > generators? > > > > > > > > > > >>>>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>>>> Cheers, > > > > > > > > > > >>>>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>>>> Konstantin > > > > > > > > > > >>>>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>>>> Am So., 5. Juni 2022 um 17:48 Uhr schrieb > > > Piotr > > > > > > > > > > >>> Nowojski > > > > > > > > > > >>>> < > > > > > > > > > > >>>>>>>>>>>>>> pnowoj...@apache.org>: > > > > > > > > > > >>>>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>>>>> Also +1 to what David has written. But it > > > > doesn't > > > > > > > > > > >>> mean > > > > > > > > > > >>>> we > > > > > > > > > > >>>>>>>> should > > > > > > > > > > >>>>>>>>> be > > > > > > > > > > >>>>>>>>>>>>>> waiting > > > > > > > > > > >>>>>>>>>>>>>>> indefinitely to deprecate SourceFunction. > > > > > > > > > > >>>>>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>>>>> Best, > > > > > > > > > > >>>>>>>>>>>>>>> Piotrek > > > > > > > > > > >>>>>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>>>>> niedz., 5 cze 2022 o 16:46 Jark Wu < > > > > > > > > > > >> imj...@gmail.com > > > > > > > > > > >>>> > > > > > > > > > > >>>>>>>>> napisał(a): > > > > > > > > > > >>>>>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>>>>>> +1 to David's point. > > > > > > > > > > >>>>>>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>>>>>> Usually, when we deprecate some > > interfaces, > > > we > > > > > > > > > > >>> should > > > > > > > > > > >>>>>> point > > > > > > > > > > >>>>>>>>> users > > > > > > > > > > >>>>>>>>>>>>> to > > > > > > > > > > >>>>>>>>>>>>>> use > > > > > > > > > > >>>>>>>>>>>>>>>> the recommended alternatives. > > > > > > > > > > >>>>>>>>>>>>>>>> However, implementing the new Source > > > interface > > > > > > > > > > >> for > > > > > > > > > > >>>> some > > > > > > > > > > >>>>>>>> simple > > > > > > > > > > >>>>>>>>>>>>>> scenarios > > > > > > > > > > >>>>>>>>>>>>>>> is > > > > > > > > > > >>>>>>>>>>>>>>>> too challenging and complex. > > > > > > > > > > >>>>>>>>>>>>>>>> We also found it isn't easy to push the > > > > internal > > > > > > > > > > >>>>>> connector > > > > > > > > > > >>>>>>> to > > > > > > > > > > >>>>>>>>>>>>> upgrade > > > > > > > > > > >>>>>>>>>>>>>> to > > > > > > > > > > >>>>>>>>>>>>>>>> the new Source because > > > > > > > > > > >>>>>>>>>>>>>>>> "FLIP-27 are hard to understand, while > > > > > > > > > > >>> SourceFunction > > > > > > > > > > >>>>> is > > > > > > > > > > >>>>>>>> easy". > > > > > > > > > > >>>>>>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>>>>>> +1 to make implementing a simple Source > > > easier > > > > > > > > > > >>> before > > > > > > > > > > >>>>>>>>> deprecating > > > > > > > > > > >>>>>>>>>>>>>>>> SourceFunction. > > > > > > > > > > >>>>>>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>>>>>> Best, > > > > > > > > > > >>>>>>>>>>>>>>>> Jark > > > > > > > > > > >>>>>>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>>>>>> On Sun, 5 Jun 2022 at 07:29, Jingsong > Lee > > < > > > > > > > > > > >>>>>>>>>> lzljs3620...@apache.org > > > > > > > > > > >>>>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>>>>> wrote: > > > > > > > > > > >>>>>>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>>>>>>> +1 to David and Ingo. > > > > > > > > > > >>>>>>>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>>>>>>> Before deprecate and remove > > SourceFunction, > > > > we > > > > > > > > > > >>>> should > > > > > > > > > > >>>>>>> have > > > > > > > > > > >>>>>>>>> some > > > > > > > > > > >>>>>>>>>>>>>> easier > > > > > > > > > > >>>>>>>>>>>>>>>> APIs > > > > > > > > > > >>>>>>>>>>>>>>>>> to wrap new Source, the cost to write a > > new > > > > > > > > > > >>> Source > > > > > > > > > > >>>> is > > > > > > > > > > >>>>>> too > > > > > > > > > > >>>>>>>>> high > > > > > > > > > > >>>>>>>>>>>>> now. > > > > > > > > > > >>>>>>>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>>>>>>> Ingo Bürk <airbla...@apache.org > > >于2022年6月5日 > > > > > > > > > > >>>>> 周日05:32写道： > > > > > > > > > > >>>>>>>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>>>>>>>> I +1 everything David said. The new > > Source > > > > > > > > > > >> API > > > > > > > > > > >>>>> raised > > > > > > > > > > >>>>>>> the > > > > > > > > > > >>>>>>>>>>>>>> complexity > > > > > > > > > > >>>>>>>>>>>>>>>>>> significantly. It's great to have > such a > > > > > > > > > > >> rich, > > > > > > > > > > >>>>>> powerful > > > > > > > > > > >>>>>>>> API > > > > > > > > > > >>>>>>>>>>>>> that > > > > > > > > > > >>>>>>>>>>>>>> can > > > > > > > > > > >>>>>>>>>>>>>>> do > > > > > > > > > > >>>>>>>>>>>>>>>>>> everything, but in the process we lost > > the > > > > > > > > > > >>>> ability > > > > > > > > > > >>>>> to > > > > > > > > > > >>>>>>>>> onboard > > > > > > > > > > >>>>>>>>>>>>>> people > > > > > > > > > > >>>>>>>>>>>>>>> to > > > > > > > > > > >>>>>>>>>>>>>>>>>> the APIs. > > > > > > > > > > >>>>>>>>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>>>>>>>> Best > > > > > > > > > > >>>>>>>>>>>>>>>>>> Ingo > > > > > > > > > > >>>>>>>>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>>>>>>>> On 04.06.22 21:21, David Anderson > wrote: > > > > > > > > > > >>>>>>>>>>>>>>>>>>> I'm in favor of this, but I think we > > need > > > > > > > > > > >> to > > > > > > > > > > >>>> make > > > > > > > > > > >>>>>> it > > > > > > > > > > >>>>>>>>> easier > > > > > > > > > > >>>>>>>>>>>>> to > > > > > > > > > > >>>>>>>>>>>>>>>>> implement > > > > > > > > > > >>>>>>>>>>>>>>>>>>> data generators and test sources. As > > > things > > > > > > > > > > >>>> stand > > > > > > > > > > >>>>>> in > > > > > > > > > > >>>>>>>>> 1.15, > > > > > > > > > > >>>>>>>>>>>>> unless > > > > > > > > > > >>>>>>>>>>>>>>> you > > > > > > > > > > >>>>>>>>>>>>>>>>> can > > > > > > > > > > >>>>>>>>>>>>>>>>>>> be satisfied with using a > > > > > > > > > > >>> NumberSequenceSource > > > > > > > > > > >>>>>>> followed > > > > > > > > > > >>>>>>>>> by > > > > > > > > > > >>>>>>>>>> a > > > > > > > > > > >>>>>>>>>>>>> map, > > > > > > > > > > >>>>>>>>>>>>>>>>> things > > > > > > > > > > >>>>>>>>>>>>>>>>>>> get quite complicated. I looked into > > > > > > > > > > >>> reworking > > > > > > > > > > >>>>> the > > > > > > > > > > >>>>>>> data > > > > > > > > > > >>>>>>>>>>>>>> generators > > > > > > > > > > >>>>>>>>>>>>>>>> used > > > > > > > > > > >>>>>>>>>>>>>>>>>> in > > > > > > > > > > >>>>>>>>>>>>>>>>>>> the training exercises, and got > > > discouraged > > > > > > > > > > >>> by > > > > > > > > > > >>>>> the > > > > > > > > > > >>>>>>>> amount > > > > > > > > > > >>>>>>>>>> of > > > > > > > > > > >>>>>>>>>>>>> work > > > > > > > > > > >>>>>>>>>>>>>>>>>> involved. > > > > > > > > > > >>>>>>>>>>>>>>>>>>> (The sources used in the training > want > > to > > > > > > > > > > >> be > > > > > > > > > > >>>>>>> unbounded, > > > > > > > > > > >>>>>>>>> and > > > > > > > > > > >>>>>>>>>>>>> need > > > > > > > > > > >>>>>>>>>>>>>>>>>>> watermarking in the sources, which > > means > > > > > > > > > > >> that > > > > > > > > > > >>>>> using > > > > > > > > > > >>>>>>>>>>>>>>>>> NumberSequenceSource > > > > > > > > > > >>>>>>>>>>>>>>>>>>> isn't an option.) > > > > > > > > > > >>>>>>>>>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>>>>>>>>> I think the proposed deprecation will > > be > > > > > > > > > > >>> better > > > > > > > > > > >>>>>>>> received > > > > > > > > > > >>>>>>>>> if > > > > > > > > > > >>>>>>>>>>>>> it > > > > > > > > > > >>>>>>>>>>>>>> can > > > > > > > > > > >>>>>>>>>>>>>>> be > > > > > > > > > > >>>>>>>>>>>>>>>>>>> accompanied by something that makes > > > > > > > > > > >>>> implementing > > > > > > > > > > >>>>> a > > > > > > > > > > >>>>>>>> simple > > > > > > > > > > >>>>>>>>>>>>> Source > > > > > > > > > > >>>>>>>>>>>>>>>> easier > > > > > > > > > > >>>>>>>>>>>>>>>>>>> than it is now. People are continuing > > to > > > > > > > > > > >>>>> implement > > > > > > > > > > >>>>>>> new > > > > > > > > > > >>>>>>>>>>>>>>>> SourceFunctions > > > > > > > > > > >>>>>>>>>>>>>>>>>>> because the interfaces defined by > > FLIP-27 > > > > > > > > > > >> are > > > > > > > > > > >>>>> hard > > > > > > > > > > >>>>>> to > > > > > > > > > > >>>>>>>>>>>>> understand, > > > > > > > > > > >>>>>>>>>>>>>>>> while > > > > > > > > > > >>>>>>>>>>>>>>>>>>> SourceFunction is easy. Alex, I > believe > > > you > > > > > > > > > > >>>> were > > > > > > > > > > >>>>>>>> looking > > > > > > > > > > >>>>>>>>>> into > > > > > > > > > > >>>>>>>>>>>>>>>>>> implementing > > > > > > > > > > >>>>>>>>>>>>>>>>>>> an easier-to-use building block that > > > could > > > > > > > > > > >> be > > > > > > > > > > >>>>> used > > > > > > > > > > >>>>>> in > > > > > > > > > > >>>>>>>>>>>>> situations > > > > > > > > > > >>>>>>>>>>>>>>> like > > > > > > > > > > >>>>>>>>>>>>>>>>>> this. > > > > > > > > > > >>>>>>>>>>>>>>>>>>> Can we get something like that in > place > > > > > > > > > > >>> first? > > > > > > > > > > >>>>>>>>>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>>>>>>>>> David > > > > > > > > > > >>>>>>>>>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>>>>>>>>> On Fri, Jun 3, 2022 at 4:52 PM Jing > Ge > > < > > > > > > > > > > >>>>>>>>> j...@ververica.com > > > > > > > > > > >>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>>>> wrote: > > > > > > > > > > >>>>>>>>>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>>>>>>>>>> Hi, > > > > > > > > > > >>>>>>>>>>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>>>>>>>>>> Thanks Alex for driving this! > > > > > > > > > > >>>>>>>>>>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>>>>>>>>>> +1 To give the Flink developers, > > > > > > > > > > >> especially > > > > > > > > > > >>>>>>> Connector > > > > > > > > > > >>>>>>>>>>>>> developers > > > > > > > > > > >>>>>>>>>>>>>>> the > > > > > > > > > > >>>>>>>>>>>>>>>>>> clear > > > > > > > > > > >>>>>>>>>>>>>>>>>>>> signal that the new Source API is > > > > > > > > > > >>> recommended > > > > > > > > > > >>>>>>>> according > > > > > > > > > > >>>>>>>>> to > > > > > > > > > > >>>>>>>>>>>>>>> FLIP-27, > > > > > > > > > > >>>>>>>>>>>>>>>> we > > > > > > > > > > >>>>>>>>>>>>>>>>>>>> should mark them as deprecated. > > > > > > > > > > >>>>>>>>>>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>>>>>>>>>> There are some open questions to > > > discuss: > > > > > > > > > > >>>>>>>>>>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>>>>>>>>>> 1. Do we need to mark all > > > > > > > > > > >>>>> subinterfaces/subclasses > > > > > > > > > > >>>>>>> as > > > > > > > > > > >>>>>>>>>>>>>> deprecated? > > > > > > > > > > >>>>>>>>>>>>>>>> e.g. > > > > > > > > > > >>>>>>>>>>>>>>>>>>>> FromElementsFunction, etc. there are > > > many. > > > > > > > > > > >>>> What > > > > > > > > > > >>>>>> are > > > > > > > > > > >>>>>>>> the > > > > > > > > > > >>>>>>>>>>>>>>>> replacements? > > > > > > > > > > >>>>>>>>>>>>>>>>>>>> 2. Do we need to mark all subclasses > > > that > > > > > > > > > > >>> have > > > > > > > > > > >>>>>>>>> replacement > > > > > > > > > > >>>>>>>>>>>>> as > > > > > > > > > > >>>>>>>>>>>>>>>>>> deprecated? > > > > > > > > > > >>>>>>>>>>>>>>>>>>>> e.g. ExternallyInducedSource whose > > > > > > > > > > >>> replacement > > > > > > > > > > >>>>>>> class, > > > > > > > > > > >>>>>>>>> if I > > > > > > > > > > >>>>>>>>>>>>> am > > > > > > > > > > >>>>>>>>>>>>>> not > > > > > > > > > > >>>>>>>>>>>>>>>>>> mistaken, > > > > > > > > > > >>>>>>>>>>>>>>>>>>>> ExternallyInducedSourceReader is > > > > > > > > > > >>> @Experimental > > > > > > > > > > >>>>>>>>>>>>>>>>>>>> 3. Do we need to mark all related > test > > > > > > > > > > >>> utility > > > > > > > > > > >>>>>>> classes > > > > > > > > > > >>>>>>>>> as > > > > > > > > > > >>>>>>>>>>>>>>>> deprecated? > > > > > > > > > > >>>>>>>>>>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>>>>>>>>>> I think it might make sense to > create > > an > > > > > > > > > > >>>>> umbrella > > > > > > > > > > >>>>>>>> ticket > > > > > > > > > > >>>>>>>>>> to > > > > > > > > > > >>>>>>>>>>>>>> cover > > > > > > > > > > >>>>>>>>>>>>>>>> all > > > > > > > > > > >>>>>>>>>>>>>>>>> of > > > > > > > > > > >>>>>>>>>>>>>>>>>>>> these with the following process: > > > > > > > > > > >>>>>>>>>>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>>>>>>>>>> 1. Mark SourceFunction as deprecated > > > asap. > > > > > > > > > > >>>>>>>>>>>>>>>>>>>> 2. Mark subinterfaces and subclasses > > as > > > > > > > > > > >>>>>> deprecated, > > > > > > > > > > >>>>>>> if > > > > > > > > > > >>>>>>>>>>>>> there are > > > > > > > > > > >>>>>>>>>>>>>>>>>> graduated > > > > > > > > > > >>>>>>>>>>>>>>>>>>>> replacements. Good example is that > > > > > > > > > > >>> KafkaSource > > > > > > > > > > >>>>>>>> replaced > > > > > > > > > > >>>>>>>>>>>>>>>> KafkaConsumer > > > > > > > > > > >>>>>>>>>>>>>>>>>> which > > > > > > > > > > >>>>>>>>>>>>>>>>>>>> has been marked as deprecated. > > > > > > > > > > >>>>>>>>>>>>>>>>>>>> 3. Do not mark subinterfaces and > > > > > > > > > > >> subclasses > > > > > > > > > > >>> as > > > > > > > > > > >>>>>>>>> deprecated, > > > > > > > > > > >>>>>>>>>>>>> if > > > > > > > > > > >>>>>>>>>>>>>>>>>> replacement > > > > > > > > > > >>>>>>>>>>>>>>>>>>>> classes are still experimental, > check > > if > > > > > > > > > > >> it > > > > > > > > > > >>> is > > > > > > > > > > >>>>>> time > > > > > > > > > > >>>>>>> to > > > > > > > > > > >>>>>>>>>>>>> graduate > > > > > > > > > > >>>>>>>>>>>>>>>> them. > > > > > > > > > > >>>>>>>>>>>>>>>>>> After > > > > > > > > > > >>>>>>>>>>>>>>>>>>>> graduation, go to step 2. It might > > take > > > a > > > > > > > > > > >>>> while > > > > > > > > > > >>>>>> for > > > > > > > > > > >>>>>>>>>>>>> graduation. > > > > > > > > > > >>>>>>>>>>>>>>>>>>>> 4. Do not mark subinterfaces and > > > > > > > > > > >> subclasses > > > > > > > > > > >>> as > > > > > > > > > > >>>>>>>>> deprecated, > > > > > > > > > > >>>>>>>>>>>>> if > > > > > > > > > > >>>>>>>>>>>>>> the > > > > > > > > > > >>>>>>>>>>>>>>>>>>>> replacement classes are experimental > > and > > > > > > > > > > >> are > > > > > > > > > > >>>> too > > > > > > > > > > >>>>>>> young > > > > > > > > > > >>>>>>>>> to > > > > > > > > > > >>>>>>>>>>>>>>> graduate. > > > > > > > > > > >>>>>>>>>>>>>>>> We > > > > > > > > > > >>>>>>>>>>>>>>>>>> have > > > > > > > > > > >>>>>>>>>>>>>>>>>>>> to wait. But in this case we could > > > create > > > > > > > > > > >>> new > > > > > > > > > > >>>>>>> tickets > > > > > > > > > > >>>>>>>>>> under > > > > > > > > > > >>>>>>>>>>>>> the > > > > > > > > > > >>>>>>>>>>>>>>>>> umbrella > > > > > > > > > > >>>>>>>>>>>>>>>>>>>> ticket. > > > > > > > > > > >>>>>>>>>>>>>>>>>>>> 5. Do not mark subinterfaces and > > > > > > > > > > >> subclasses > > > > > > > > > > >>> as > > > > > > > > > > >>>>>>>>> deprecated, > > > > > > > > > > >>>>>>>>>>>>> if > > > > > > > > > > >>>>>>>>>>>>>>> there > > > > > > > > > > >>>>>>>>>>>>>>>> is > > > > > > > > > > >>>>>>>>>>>>>>>>>> no > > > > > > > > > > >>>>>>>>>>>>>>>>>>>> replacement at all. We have to > create > > > new > > > > > > > > > > >>>>> tickets > > > > > > > > > > >>>>>>> and > > > > > > > > > > >>>>>>>>> wait > > > > > > > > > > >>>>>>>>>>>>> until > > > > > > > > > > >>>>>>>>>>>>>>> the > > > > > > > > > > >>>>>>>>>>>>>>>>> new > > > > > > > > > > >>>>>>>>>>>>>>>>>>>> implementation has been done and > > > > > > > > > > >> graduated. > > > > > > > > > > >>> It > > > > > > > > > > >>>>>> will > > > > > > > > > > >>>>>>>>> take a > > > > > > > > > > >>>>>>>>>>>>>> longer > > > > > > > > > > >>>>>>>>>>>>>>>>> time, > > > > > > > > > > >>>>>>>>>>>>>>>>>>>> roughly 1,5 years. > > > > > > > > > > >>>>>>>>>>>>>>>>>>>> 6. For test classes, we could follow > > the > > > > > > > > > > >>> same > > > > > > > > > > >>>>>> rule. > > > > > > > > > > >>>>>>>> But > > > > > > > > > > >>>>>>>>> I > > > > > > > > > > >>>>>>>>>>>>> think > > > > > > > > > > >>>>>>>>>>>>>>> for > > > > > > > > > > >>>>>>>>>>>>>>>>> some > > > > > > > > > > >>>>>>>>>>>>>>>>>>>> cases, we could consider doing the > > > > > > > > > > >>> replacement > > > > > > > > > > >>>>>>>> directly > > > > > > > > > > >>>>>>>>>>>>> without > > > > > > > > > > >>>>>>>>>>>>>>>> going > > > > > > > > > > >>>>>>>>>>>>>>>>>>>> through the deprecation phase. > > > > > > > > > > >>>>>>>>>>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>>>>>>>>>> When we look back on all of these, > we > > > can > > > > > > > > > > >>>>> realize > > > > > > > > > > >>>>>> it > > > > > > > > > > >>>>>>>> is > > > > > > > > > > >>>>>>>>> a > > > > > > > > > > >>>>>>>>>>>>> big > > > > > > > > > > >>>>>>>>>>>>>> epic > > > > > > > > > > >>>>>>>>>>>>>>>>> (even > > > > > > > > > > >>>>>>>>>>>>>>>>>>>> bigger than an epic). It needs > someone > > > to > > > > > > > > > > >>>> drive > > > > > > > > > > >>>>> it > > > > > > > > > > >>>>>>> and > > > > > > > > > > >>>>>>>>>> keep > > > > > > > > > > >>>>>>>>>>>>>> focus > > > > > > > > > > >>>>>>>>>>>>>>> on > > > > > > > > > > >>>>>>>>>>>>>>>>> it > > > > > > > > > > >>>>>>>>>>>>>>>>>>>> continuously with support from the > > > > > > > > > > >> community > > > > > > > > > > >>>> and > > > > > > > > > > >>>>>>> push > > > > > > > > > > >>>>>>>>> the > > > > > > > > > > >>>>>>>>>>>>>>>> development > > > > > > > > > > >>>>>>>>>>>>>>>>>>>> towards the new Source API of > FLIP-27. > > > > > > > > > > >>>>>>>>>>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>>>>>>>>>> If we could have consensus for this, > > > Alex > > > > > > > > > > >>>> and I > > > > > > > > > > >>>>>>> could > > > > > > > > > > >>>>>>>>>>>>> create > > > > > > > > > > >>>>>>>>>>>>>> the > > > > > > > > > > >>>>>>>>>>>>>>>>>> umbrella > > > > > > > > > > >>>>>>>>>>>>>>>>>>>> ticket to kick it off. > > > > > > > > > > >>>>>>>>>>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>>>>>>>>>> Best regards, > > > > > > > > > > >>>>>>>>>>>>>>>>>>>> Jing > > > > > > > > > > >>>>>>>>>>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>>>>>>>>>> On Fri, Jun 3, 2022 at 3:54 PM > > Alexander > > > > > > > > > > >>>>> Fedulov < > > > > > > > > > > >>>>>>>>>>>>>>>>>> alexan...@ververica.com> > > > > > > > > > > >>>>>>>>>>>>>>>>>>>> wrote: > > > > > > > > > > >>>>>>>>>>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>>>>>>>>>>> Hi everyone, > > > > > > > > > > >>>>>>>>>>>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>>>>>>>>>>> I would like to start the > discussion > > > > > > > > > > >> about > > > > > > > > > > >>>>>> marking > > > > > > > > > > >>>>>>>>>>>>>>>>> SourceFunction-based > > > > > > > > > > >>>>>>>>>>>>>>>>>>>>> interfaces as deprecated. With the > > > > > > > > > > >> FLIP-27 > > > > > > > > > > >>>> APIs > > > > > > > > > > >>>>>>>>> becoming > > > > > > > > > > >>>>>>>>>>>>> the > > > > > > > > > > >>>>>>>>>>>>>> new > > > > > > > > > > >>>>>>>>>>>>>>>>>>>> standard, > > > > > > > > > > >>>>>>>>>>>>>>>>>>>>> the old ones have to be eventually > > > phased > > > > > > > > > > >>>> out. > > > > > > > > > > >>>>>>>> Although > > > > > > > > > > >>>>>>>>>>>>> this > > > > > > > > > > >>>>>>>>>>>>>>> state > > > > > > > > > > >>>>>>>>>>>>>>>> is > > > > > > > > > > >>>>>>>>>>>>>>>>>>>> well > > > > > > > > > > >>>>>>>>>>>>>>>>>>>>> known within the community and no > new > > > > > > > > > > >>>>> connectors > > > > > > > > > > >>>>>>>> based > > > > > > > > > > >>>>>>>>> on > > > > > > > > > > >>>>>>>>>>>>> the > > > > > > > > > > >>>>>>>>>>>>>> old > > > > > > > > > > >>>>>>>>>>>>>>>>>>>>> interfaces can be accepted into the > > > > > > > > > > >>> project, > > > > > > > > > > >>>>> the > > > > > > > > > > >>>>>>>>>> footprint > > > > > > > > > > >>>>>>>>>>>>> of > > > > > > > > > > >>>>>>>>>>>>>>>>>>>>> SourceFunction in the user code > still > > > > > > > > > > >> keeps > > > > > > > > > > >>>>>> growing > > > > > > > > > > >>>>>>>>>>>>> (primarily > > > > > > > > > > >>>>>>>>>>>>>>> for > > > > > > > > > > >>>>>>>>>>>>>>>>> data > > > > > > > > > > >>>>>>>>>>>>>>>>>>>>> generators and test utilities). I > > > believe > > > > > > > > > > >>> it > > > > > > > > > > >>>> is > > > > > > > > > > >>>>>>> best > > > > > > > > > > >>>>>>>> to > > > > > > > > > > >>>>>>>>>>>>> mark > > > > > > > > > > >>>>>>>>>>>>>>>>>>>> SourceFunction > > > > > > > > > > >>>>>>>>>>>>>>>>>>>>> as deprecated as soon as possible. > > What > > > > > > > > > > >> do > > > > > > > > > > >>>> you > > > > > > > > > > >>>>>>> think? > > > > > > > > > > >>>>>>>>>>>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>>>>>>>>>>> Best, > > > > > > > > > > >>>>>>>>>>>>>>>>>>>>> Alexander Fedulov > > > > > > > > > > >>>>>>>>>>>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>>>> -- > > > > > > > > > > >>>>>>>>>>>>>> https://twitter.com/snntrable > > > > > > > > > > >>>>>>>>>>>>>> https://github.com/knaufk > > > > > > > > > > >>>>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>> > > > > > > > > > > >>>>>>>>> > > > > > > > > > > >>>>>>>> > > > > > > > > > > >>>>>>> > > > > > > > > > > >>>>>> > > > > > > > > > > >>>>> > > > > > > > > > > >>>> > > > > > > > > > > >>> > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > https://twitter.com/snntrable > > > > > > > > https://github.com/knaufk > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
