Hi Michael, Unfortunately the new `KinesisDataStreamsSink` doesn't support aggregation yet. My suggestion if you want to use native kinesis aggregation is to use the latest connector version that supports KPL as sink for Table API, that would be 1.14.x. you could package the connector of that version.
> was thinking about moving the logic to use window functions (either in the Table or DataStream API), but here I think I'd need to "close" the window based not only on time, but also on record number Regarding this approach, I believe a better way might be implementing a custom process function to hold batches of records in state and emit an aggregated record but this might not be consistent with KPL aggregation of course and de-aggregated records could be not retrieved so I would advise not to take this approach. Best Regards Ahmed Hamdy On Mon, 29 Apr 2024 at 11:14, Michael Marino <michael.mar...@tado.com> wrote: > Hi all, > > We are currently using Flink 1.18.1 (AWS Managed Flink) and are writing to > Kinesis streams in several of our applications using the Table API. > > In our use case, we would like to be able to aggregate multiple records > (rows) together and emit them in a single Kinesis record. > > As far as I understand, with the usage of the Async Writer it is not > possible to aggregate records (Table rows) together into a single record as > was possible previously with the Kinesis Producer Library. > > I wanted to ask if anyone here has any suggestions of how to do this, or > perhaps if I missed it somewhere in the documentation? I was thinking about > moving the logic to use window functions (either in the Table or DataStream > API), but here I think I'd need to "close" the window based not only on > time, but also on record number. Anyways, any thoughts are appreciated! > > Thanks, > Mike > > -- > > Michael Marino > > Principal Data Science & Analytics > > Phone: +49 89 7167786 - 14 > > linkedin.com/company/tadogmbh <https://www.linkedin.com/company/tadogmbh> > | facebook.com/tado <http://www.facebook.com/tado> | twitter.com/tado > <http://www.twitter.com/tado> | youtube.com/tado > <http://www.youtube.com/tado> > > www.tado.com | tado GmbH | Sapporobogen 6-8 | 80637 Munich | Germany > > Managing Directors: Dr. Philip Beckmann | Christian Deilmann | Johannes > Schwarz | Josef Wenzl > > Registered with the Commercial Register Munich as HRB 194769 | VAT-No: DE > 280012558 >
