I +1 everything David said. The new Source API raised the complexity
significantly. It's great to have such a rich, powerful API that can do
everything, but in the process we lost the ability to onboard people to
the APIs.
Best
Ingo
On 04.06.22 21:21, David Anderson wrote:
I'm in favor of this, but I think we need to make it easier to implement
data generators and test sources. As things stand in 1.15, unless you can
be satisfied with using a NumberSequenceSource followed by a map, things
get quite complicated. I looked into reworking the data generators used in
the training exercises, and got discouraged by the amount of work involved.
(The sources used in the training want to be unbounded, and need
watermarking in the sources, which means that using NumberSequenceSource
isn't an option.)
I think the proposed deprecation will be better received if it can be
accompanied by something that makes implementing a simple Source easier
than it is now. People are continuing to implement new SourceFunctions
because the interfaces defined by FLIP-27 are hard to understand, while
SourceFunction is easy. Alex, I believe you were looking into implementing
an easier-to-use building block that could be used in situations like this.
Can we get something like that in place first?
David
On Fri, Jun 3, 2022 at 4:52 PM Jing Ge <j...@ververica.com> wrote:
Hi,
Thanks Alex for driving this!
+1 To give the Flink developers, especially Connector developers the clear
signal that the new Source API is recommended according to FLIP-27, we
should mark them as deprecated.
There are some open questions to discuss:
1. Do we need to mark all subinterfaces/subclasses as deprecated? e.g.
FromElementsFunction, etc. there are many. What are the replacements?
2. Do we need to mark all subclasses that have replacement as deprecated?
e.g. ExternallyInducedSource whose replacement class, if I am not mistaken,
ExternallyInducedSourceReader is @Experimental
3. Do we need to mark all related test utility classes as deprecated?
I think it might make sense to create an umbrella ticket to cover all of
these with the following process:
1. Mark SourceFunction as deprecated asap.
2. Mark subinterfaces and subclasses as deprecated, if there are graduated
replacements. Good example is that KafkaSource replaced KafkaConsumer which
has been marked as deprecated.
3. Do not mark subinterfaces and subclasses as deprecated, if replacement
classes are still experimental, check if it is time to graduate them. After
graduation, go to step 2. It might take a while for graduation.
4. Do not mark subinterfaces and subclasses as deprecated, if the
replacement classes are experimental and are too young to graduate. We have
to wait. But in this case we could create new tickets under the umbrella
ticket.
5. Do not mark subinterfaces and subclasses as deprecated, if there is no
replacement at all. We have to create new tickets and wait until the new
implementation has been done and graduated. It will take a longer time,
roughly 1,5 years.
6. For test classes, we could follow the same rule. But I think for some
cases, we could consider doing the replacement directly without going
through the deprecation phase.
When we look back on all of these, we can realize it is a big epic (even
bigger than an epic). It needs someone to drive it and keep focus on it
continuously with support from the community and push the development
towards the new Source API of FLIP-27.
If we could have consensus for this, Alex and I could create the umbrella
ticket to kick it off.
Best regards,
Jing
On Fri, Jun 3, 2022 at 3:54 PM Alexander Fedulov <alexan...@ververica.com>
wrote:
Hi everyone,
I would like to start the discussion about marking SourceFunction-based
interfaces as deprecated. With the FLIP-27 APIs becoming the new
standard,
the old ones have to be eventually phased out. Although this state is
well
known within the community and no new connectors based on the old
interfaces can be accepted into the project, the footprint of
SourceFunction in the user code still keeps growing (primarily for data
generators and test utilities). I believe it is best to mark
SourceFunction
as deprecated as soon as possible. What do you think?
Best,
Alexander Fedulov
