Hi Aleks, Thanks for the valuable feedback, yeah I agree it might be confusing for some users. Best Regards Ahmed Hamdy
On Sun, 24 Mar 2024 at 18:40, Aleksandr Pilipenko <z3d...@gmail.com> wrote: > Hi Ahmed, thank you for starting this discussion. > > In general, I am more inclined toward proposal 2 - having logic that > dynamically selects implementation under the hood might introduce > unexpected behaviors for users, considering that one of the options is an > experimental connector with an unstable API. > > Best regards, > Aleksandr Pilipenko > > On Sun, 24 Mar 2024 at 15:19, Ahmed Hamdy <hamdy10...@gmail.com> wrote: > > > Hi devs, > > With the delivery of FLINK-31813 > > <https://issues.apache.org/jira/browse/FLINK-31813>[1] that offers a new > > Datastream Kinesis Streams source paving the way to deprecate the KCL > > backed `FlinkKinesisConsumer`, I would like to start working on > migrating > > Kinesis Source Table API to the new source (FLINK-31987 > > <https://issues.apache.org/jira/browse/FLINK-31987>[2]) in a > > similar approach to what I did with the sink migration. This work however > > is blocked on migration of additional features like EFO support and > > deaggregation which seems to be idle for some time now. I want to discuss > > an alternative approach for migration ensuring backward compatibility. > > > > Proposal 1: Given the pre-existing dependency from > > `flink-connector-kinesis` to `flink-connector-aws-kinesis-streams` I > > suggest we implement a `KinesisStreamsDynamicSource` backed by the new > > `KinesisStreamsSource` and add a `KinesisSourceSelector` to the existing > > source factory to dynamically select a source according to features > enabled > > from the DDL, i.e shall the unsupported features like EFO or aggregation > be > > enabled the legacy dynamic source would be provided, otherwise the new > > dynamic source is to be provided. Once all other features are implemented > > we migrate the factory as well. > > Proposal 2: We do nothing and wait for all features to be implemented. > > Let me know your thoughts. > > > > 1- https://issues.apache.org/jira/browse/FLINK-31813 > > 2- https://issues.apache.org/jira/browse/FLINK-31987 > > Best Regards > > Ahmed Hamdy > > >
