Hi Lorenzo, I am not sure I follow the issue of awaiting a major release, we have seen the major releases for AWS connectors being picked up quite fast so we can just push for one. Plus the contribution is not a bug fix, additionally the legacy source is not yet removed. hence I would personally not rush any workaround in the apache AWS connectors repo but rather push for a release once the feature is merged. Best Regards Ahmed Hamdy
On Mon, 28 Apr 2025 at 17:22, Lorenzo Nicora <lorenzo.nic...@gmail.com> wrote: > Hello > the current version of Kinesis source (5.0.0) lacks support for KLP > de-aggregation [1] > There is a pending PR [2] but it breaks the API and will require a new > major version of the connector. > > I implemented an alternative solution using a DeserializationSchema wrapper > which adds de-aggregation to any other DeserializationSchema [3]. > > Would it be a reasonable approach including this DeserializationSchema > wrapper (with proper tests) as part of the connector distribution? > A user can decide to use the wrapper if they need de-aggregation. > The wrapper works fine even if the stream is not aggregated, with some > minimal overhead. > > The wrapper will be redundant when/if the main fix [1] is released. > However, requiring a new major version, this may take a while and some > users are blocked from upgrading to the new source by lack of > de-aggregation support. > > Any opinion on this approach? > > Lorenzo > > [1] https://issues.apache.org/jira/browse/FLINK-32097 > [2] https://github.com/apache/flink-connector-aws/pull/188 > [3] > > https://github.com/aws-samples/amazon-managed-service-for-apache-flink-examples/blob/main/java/KinesisSourceDeaggregation/flink-app/src/main/java/com/amazonaws/services/msf/deaggregation/KinesisDeaggregatingDeserializationSchemaWrapper.java >
