In Iceberg source, we have a data generator source that can control the records per checkpoint cycle. Can we support sth like this in the DataGeneratorSource?
https://github.com/apache/iceberg/blob/master/flink/v1.15/flink/src/test/java/org/apache/iceberg/flink/source/BoundedTestSource.java public BoundedTestSource(List<List<T>> elementsPerCheckpoint, boolean checkpointEnabled) Thanks, Steven On Tue, Jun 7, 2022 at 8:48 AM Alexander Fedulov <alexan...@ververica.com> wrote: > Hi everyone, > > I would like to open a discussion on FLIP-238: Introduce FLIP-27-based Data > Generator Source [1]. During the discussion about deprecating the > SourceFunction API [2] it became evident that an easy-to-use > FLIP-27-compatible data generator source is needed so that the current > SourceFunction-based data generator implementations could be phased out for > both Flink demo/PoC applications and for the internal Flink tests. This > FLIP proposes to introduce a generic DataGeneratorSource capable of > producing events of an arbitrary type based on a user-supplied MapFunction. > > Looking forward to your feedback. > > [1] https://cwiki.apache.org/confluence/x/9Av1D > [2] https://lists.apache.org/thread/d6cwqw9b3105wcpdkwq7rr4s7x4ywqr9 > > Best, > Alexander Fedulov >
