Hi all,
I've tried to migrate my very simple Elasticsearch SourceFunction (that
use scroll API and produce batch of documents) to new Source API, but I
gave up because it's too complicated. It should much simpler to migrate
that function to a bounded or unbounded source.
Before removing completely
> do we have any plan to offer a lighter Source API to decrease the connector
> development cost?
I remember mentioning it many times, but no contributor did it. ToT
Best,
Jingsong
On Tue, Jul 4, 2023 at 11:01 AM Leonard Xu wrote:
>
> +1 to deprecate.
> +1 for David’s points.
>
> I’ve one rela
+1 to deprecate.
+1 for David’s points.
I’ve one related question, do we have any plan to offer a lighter Source API to
decrease the connector development cost?
New Source API is good but too heavy for use cases like tests or even some
simple connectors.
Best,
Leonard
> On Jun 6, 2022, at 9:
I am happy to announce that the blocker has been resolved and SourceFunction
is now marked as @Deprecated [1].
The work continues to remove the dependencies on the SourceFunction API in
Flink internals in order to prepare for dropping it completely in Flink 2.0.
I'd like to get some opinions on a
I'd like to revive the efforts to deprecate the SourceFunction API.
It would be great to get a review for this PR:
https://github.com/apache/flink/pull/21774
It immediately unblocks marking the actual SourceFunction as deprecated.
https://github.com/apache/flink/pull/20049
There is also this wor
Thank you all for your valuable input and participation in the discussion
The vote is open now [1]
[1] https://lists.apache.org/thread/kv9rj3w2rmkb8jtss5bqffhw57or7v8v
Best,
Alexander Fedulov
On Tue, Jun 14, 2022 at 10:21 PM Jing Ge wrote:
> Hi Martijn,
>
> Yes, that is exactly what I trie
Hi Martijn,
Yes, that is exactly what I tried to say implicitly in a polite manner. :))
Best regards,
Jing
On Tue, Jun 14, 2022 at 8:12 PM Martijn Visser
wrote:
> Hi Alex and Jing,
>
> According to the Flink Bylaws and FLIP guide, this change would be a
> "change that impacts the public inter
Hi Alex and Jing,
According to the Flink Bylaws and FLIP guide, this change would be a
"change that impacts the public interfaces of the project". I do see little
value in creating a special deprecation FLIP, but I do think it requires a
[VOTE] thread which follows the FLIP approval process. So ba
Hi Jing,
I don't think we do pure deprecation FLIPs. I am also OK if we consider
consensus in this thread enough to proceed with opening the actual
deprecation PRs for the first two items of the umbrella ticket. In that
case, we can skip the vote.
What do people prefer? Do you think we need to do
Hi Alex,
I guess you are meaning to start a new voting thread following the FLIP
discussion&voting concept. Looking forward to it!
Best regards,
Jing
On Tue, Jun 14, 2022 at 3:35 PM Alexander Fedulov
wrote:
> Hi Becket,
>
> thanks for your feedback. As proposed, I started an umbrella ticket [1
Hi Becket,
thanks for your feedback. As proposed, I started an umbrella ticket [1] to
collect all the
steps needed. Please add any missing items.
Judging by the discussion on this thread I propose to open a vote on the
first two
subtasks since they are immediately actionable [2] [3].
[1] https:/
In general, I'll give a big +1 to deprecating the SourceFunction.
That said, it is indeed worth looking into what might be missing or less
easy to implement with FLIP-27 Source compared with the SourceFunction.
Maybe we can just compile a list of things to do in order to fully
deprecate the Source
+1 for deprecating SourceFunction from me as well. And a big THANK YOU to
Alex Fedulov for bringing forward FLIP-238.
David
On Fri, Jun 10, 2022 at 4:03 AM Lijie Wang wrote:
> Hi all,
>
> Sorry for my mistake. The `StreamExecutionEnvironment#readFiles` and can be
> easily replaced by `FileSourc
Hi all,
Sorry for my mistake. The `StreamExecutionEnvironment#readFiles` and can be
easily replaced by `FileSource#forRecordStreamFormat/forBulkFileFormat`. I
have no other concerns.
+1 to deprecate SourceFunction and deprecate the methods (in
StreamExecutionEnvironment) based on SourceFunction
Hi everyone,
thank you Jing for redirecting the discussion back to the topic at hand. I
agree with all of your points.
+1 to deprecate SourceFunction
Is there really no replacement for the StreamExecutionEnvironment#readXXX.
There is already a FLIP-27 based FileSource, right? What's missing to
r
Hi all,
It seems that there is some understandable cautiousness with regard to
deprecating methods and subclasses that do not have alternatives just yet.
We should probably first agree if it is in general OK for Flink to use
@Deprecated
annotation for parts of the code that do not have alternativ
Hi,
these APIs don't expose the underlying source directly, so I don't think
we need to worry about deprecating them as well. There's also nothing
inherently wrong with using a deprecated API internally, though even
just for the experience of using our own new APIs I would personally say
that
Hi Martijn,
I don't mean it's a blocker. Just a information. And I'm also +1 for this.
Put it another way: should we migrate the `#readFile(...)` to new API or
provide a similar method "readxxx“ based on the new Source API?
And if we don't migrate it, does it mean that the `#readFile(...)` shou
Hi Lijie,
I don't see any problem with deprecating those methods at this moment, as
long as we don't remove them until the replacements are available. Besides
that, are we sure there are no replacements already, especially with the
new FileSource?
Best regards,
Martijn
Op do 9 jun. 2022 om 14:2
Hi all,
FYI, currently, some commonly used methods in StreamExecutionEnvironment
are still based on the old SourceFunction (and there is no alternative):
`StreamExecutionEnvironment#readFile(...)`
`StreamExecutionEnvironment#readTextFile(...)`
I think these should be migrated to the new source AP
Hi all,
I think implicitly we've already considered the SourceFunction and
SinkFunction as deprecated. They are even marked as so on the Flink roadmap
[1]. That also shows that connectors that are using these interfaces are
either approaching end-of-life. The fact that we're actively migrating
con
Hi,
I am very happy to see opinions from different perspectives. That will help
us understand the problem better. Thanks all for the informative discussion.
Let's see the big picture and check following facts together:
1. FLIP-27 was intended to solve some technical issues that are very
difficul
Hey Austin,
Since we are getting deeper into the implementation details of the
DataGeneratorSource
and it is not the main topic of this thread, I propose to move our
discussion to where it belongs: [DISCUSS] FLIP-238 [1]. Could you please
briefly formulate your requirements to make it easier for t
> @Austin, in the FLIP I mentioned above [1], the user is expected to
pass a MapFunction
to the generator. I wonder if you could have your external client and
polling logic wrapped in a custom
MapFunction implementation class? Would that answer your needs or do you
have some
more sophisticated scen
Hi everyone,
Thanks for all the input and a lively discussion. It seems that there is a
consensus that due to
the inherent complexity of FLIP-27 sources we should provide more
user-facing utilities to bridge
the gap between the existing SourceFunction-based functionality and the new
APIs.
To star
Thanks for the nice discussion all.
I was recently trying to implement a very simple polling source and
would've loved a higher-level base to work from. I'm wondering if in
addition to the data generator use cases, it would be good to support a
simple non-parallel polling abstraction to make it ea
Well. It's a bit off-topic. For deprecating SourceFunction as FLIP-27
series works go ahead, +1 from my side. It's a significant work towards the
unification of batch and streaming effort :)
Best,
tison.
tison 于2022年6月6日周一 21:54写道：
> The starting point of the version bump and removal question
The starting point of the version bump and removal question is that
downstream projects may experience a tough time to adapt new interfaces
while Flink keeps in 1.x versions so that users may expect it as an easy
task. From my experience, it's really challenge to maintain
compatibility between mult
One question from my side:
As SourceFunction a @Public interface, we cannot remove it before doing a
major version bump (Flink 2.0).
Of course it's not a blocker to make such deprecation and let the new
interface step in. My question is whether we have a plan to finally remove
the deprecated inte
>
> David, can you elaborate why you need watermark generation in the source
> for your data generators?
The training exercises should strive to provide examples of best practices.
If the exercises and their solutions use
env.fromSource(source, WatermarkStrategy.noWatermarks(), "name-of-source")
Hi everyone,
very interesting thread. The proposal for deprecation seems to have sparked
a very important discussion. Do we what users struggle with specifically?
Speaking for myself, when I upgrade flink-faker to the new Source API an
unbounded version of the NumberSequenceSource would have been
Also +1 to what David has written. But it doesn't mean we should be waiting
indefinitely to deprecate SourceFunction.
Best,
Piotrek
niedz., 5 cze 2022 o 16:46 Jark Wu napisał(a):
> +1 to David's point.
>
> Usually, when we deprecate some interfaces, we should point users to use
> the recommende
+1 to David's point.
Usually, when we deprecate some interfaces, we should point users to use
the recommended alternatives.
However, implementing the new Source interface for some simple scenarios is
too challenging and complex.
We also found it isn't easy to push the internal connector to upgrade
+1 to David and Ingo.
Before deprecate and remove SourceFunction, we should have some easier APIs
to wrap new Source, the cost to write a new Source is too high now.
Ingo Bürk 于2022年6月5日 周日05:32写道：
> I +1 everything David said. The new Source API raised the complexity
> significantly. It's gre
I +1 everything David said. The new Source API raised the complexity
significantly. It's great to have such a rich, powerful API that can do
everything, but in the process we lost the ability to onboard people to
the APIs.
Best
Ingo
On 04.06.22 21:21, David Anderson wrote:
I'm in favor of t
I'm in favor of this, but I think we need to make it easier to implement
data generators and test sources. As things stand in 1.15, unless you can
be satisfied with using a NumberSequenceSource followed by a map, things
get quite complicated. I looked into reworking the data generators used in
the
Hi,
Thanks Alex for driving this!
+1 To give the Flink developers, especially Connector developers the clear
signal that the new Source API is recommended according to FLIP-27, we
should mark them as deprecated.
There are some open questions to discuss:
1. Do we need to mark all subinterfaces/s
Hi everyone,
I would like to start the discussion about marking SourceFunction-based
interfaces as deprecated. With the FLIP-27 APIs becoming the new standard,
the old ones have to be eventually phased out. Although this state is well
known within the community and no new connectors based on the o
38 matches
Mail list logo
