Hi Steven, FYI, I've added your requirement to the list of subtasks for deprecating the SourceFunction API [1] [2].
[1] https://issues.apache.org/jira/browse/FLINK-28045 [2] https://issues.apache.org/jira/browse/FLINK-28054 Best, Alexander Fedulov On Tue, Jun 7, 2022 at 6:03 PM Steven Wu <stevenz...@gmail.com> wrote: > In Iceberg source, we have a data generator source that can control the > records per checkpoint cycle. Can we support sth like this in the > DataGeneratorSource? > > > https://github.com/apache/iceberg/blob/master/flink/v1.15/flink/src/test/java/org/apache/iceberg/flink/source/BoundedTestSource.java > public BoundedTestSource(List<List<T>> elementsPerCheckpoint, boolean > checkpointEnabled) > > Thanks, > Steven > > On Tue, Jun 7, 2022 at 8:48 AM Alexander Fedulov <alexan...@ververica.com> > wrote: > > > Hi everyone, > > > > I would like to open a discussion on FLIP-238: Introduce FLIP-27-based > Data > > Generator Source [1]. During the discussion about deprecating the > > SourceFunction API [2] it became evident that an easy-to-use > > FLIP-27-compatible data generator source is needed so that the current > > SourceFunction-based data generator implementations could be phased out > for > > both Flink demo/PoC applications and for the internal Flink tests. This > > FLIP proposes to introduce a generic DataGeneratorSource capable of > > producing events of an arbitrary type based on a user-supplied > MapFunction. > > > > Looking forward to your feedback. > > > > [1] https://cwiki.apache.org/confluence/x/9Av1D > > [2] https://lists.apache.org/thread/d6cwqw9b3105wcpdkwq7rr4s7x4ywqr9 > > > > Best, > > Alexander Fedulov > > >
