+1 to David and Ingo. Before deprecate and remove SourceFunction, we should have some easier APIs to wrap new Source, the cost to write a new Source is too high now.
Ingo Bürk <airbla...@apache.org>于2022年6月5日 周日05:32写道： > I +1 everything David said. The new Source API raised the complexity > significantly. It's great to have such a rich, powerful API that can do > everything, but in the process we lost the ability to onboard people to > the APIs. > > > Best > Ingo > > On 04.06.22 21:21, David Anderson wrote: > > I'm in favor of this, but I think we need to make it easier to implement > > data generators and test sources. As things stand in 1.15, unless you can > > be satisfied with using a NumberSequenceSource followed by a map, things > > get quite complicated. I looked into reworking the data generators used > in > > the training exercises, and got discouraged by the amount of work > involved. > > (The sources used in the training want to be unbounded, and need > > watermarking in the sources, which means that using NumberSequenceSource > > isn't an option.) > > > > I think the proposed deprecation will be better received if it can be > > accompanied by something that makes implementing a simple Source easier > > than it is now. People are continuing to implement new SourceFunctions > > because the interfaces defined by FLIP-27 are hard to understand, while > > SourceFunction is easy. Alex, I believe you were looking into > implementing > > an easier-to-use building block that could be used in situations like > this. > > Can we get something like that in place first? > > > > David > > > > On Fri, Jun 3, 2022 at 4:52 PM Jing Ge <j...@ververica.com> wrote: > > > >> Hi, > >> > >> Thanks Alex for driving this! > >> > >> +1 To give the Flink developers, especially Connector developers the > clear > >> signal that the new Source API is recommended according to FLIP-27, we > >> should mark them as deprecated. > >> > >> There are some open questions to discuss: > >> > >> 1. Do we need to mark all subinterfaces/subclasses as deprecated? e.g. > >> FromElementsFunction, etc. there are many. What are the replacements? > >> 2. Do we need to mark all subclasses that have replacement as > deprecated? > >> e.g. ExternallyInducedSource whose replacement class, if I am not > mistaken, > >> ExternallyInducedSourceReader is @Experimental > >> 3. Do we need to mark all related test utility classes as deprecated? > >> > >> I think it might make sense to create an umbrella ticket to cover all of > >> these with the following process: > >> > >> 1. Mark SourceFunction as deprecated asap. > >> 2. Mark subinterfaces and subclasses as deprecated, if there are > graduated > >> replacements. Good example is that KafkaSource replaced KafkaConsumer > which > >> has been marked as deprecated. > >> 3. Do not mark subinterfaces and subclasses as deprecated, if > replacement > >> classes are still experimental, check if it is time to graduate them. > After > >> graduation, go to step 2. It might take a while for graduation. > >> 4. Do not mark subinterfaces and subclasses as deprecated, if the > >> replacement classes are experimental and are too young to graduate. We > have > >> to wait. But in this case we could create new tickets under the umbrella > >> ticket. > >> 5. Do not mark subinterfaces and subclasses as deprecated, if there is > no > >> replacement at all. We have to create new tickets and wait until the new > >> implementation has been done and graduated. It will take a longer time, > >> roughly 1,5 years. > >> 6. For test classes, we could follow the same rule. But I think for some > >> cases, we could consider doing the replacement directly without going > >> through the deprecation phase. > >> > >> When we look back on all of these, we can realize it is a big epic (even > >> bigger than an epic). It needs someone to drive it and keep focus on it > >> continuously with support from the community and push the development > >> towards the new Source API of FLIP-27. > >> > >> If we could have consensus for this, Alex and I could create the > umbrella > >> ticket to kick it off. > >> > >> Best regards, > >> Jing > >> > >> > >> On Fri, Jun 3, 2022 at 3:54 PM Alexander Fedulov < > alexan...@ververica.com> > >> wrote: > >> > >>> Hi everyone, > >>> > >>> I would like to start the discussion about marking SourceFunction-based > >>> interfaces as deprecated. With the FLIP-27 APIs becoming the new > >> standard, > >>> the old ones have to be eventually phased out. Although this state is > >> well > >>> known within the community and no new connectors based on the old > >>> interfaces can be accepted into the project, the footprint of > >>> SourceFunction in the user code still keeps growing (primarily for data > >>> generators and test utilities). I believe it is best to mark > >> SourceFunction > >>> as deprecated as soon as possible. What do you think? > >>> > >>> Best, > >>> Alexander Fedulov > >>> > >> > > >
