Hi, Please correct me if I'm wrong anywhere. I'm just new to Flink and trying to navigate the landscape.
Within my company, currently we're trying to develop a connector for our internal change data capture system(brooklin) for flink. We are planning to use Flink SQL as a primary API to build streaming applications. When exploring flink contracts, we noticed that there are two different flavors of APIs available in Flink for Source integration. a) Flink Table API : The Flink ScanTableSource abstractions which are currently relying upon the SourceFunction interfaces for integrating with the underlying messaging-client libraries. For instance, KafkaDynamicSource and KinesisDynamicSource are currently using the FlinkKafkaConsumer(an implementation of RichParallelSourceFunction) and KinesisConsumer(an implementation of RichParallelSourceFunction) respectively to read from the broker. b) FLIP-27 style connector implementations: There are connectors which implement SplitEnumerator and SourceReader abstractions, where the Enumerator runs with the JobMaster and the Readers runs within the TaskManager processes respectively. Questions: 1. If I want to integrate a new connector and want to use Flink SQL, then what is the recommendation? Are the users supposed to implement the RichParallelSourceFunction, CheckpointListener etc similar to FlinkKafkaConsumer and wire into the ScanTableSource API? 2. Just wondering, what is the long term plan for the ScanTablesource APIs? Are there plans for them to use and integrate with the SplitEnumerator and SourceReader abstractions? 3. If I want to offer my connector implementation to both Flink DataStream and Flink SQL APIs, then should I implement both the flavors of source APIs(SplitEnumerator/SourceReader as well as SourceFunction) in flink? I would really appreciate it if someone can help and answer the above questions. Thanks.
