Re: Event de duplication in flink with rabbitmq connector

2024-07-18 Thread Ahmed Hamdy
Yes, The current implementation doesn't leverage transactions on publish like it does for the source on acking and nacking the deliveries, you can raise a ticket to support exactly once RMQSinks within the community or implement the logic yourself. my checkpoints size is increasing. can this

Re: Event de duplication in flink with rabbitmq connector

2024-07-18 Thread Ahmed Hamdy
Hi Banu, This behavior of source is expected, the guarantee of the RMQSource is exactly once which is achieved by acknowledging envelopes on checkpoints hence the source would never re-read a message after checkpoint even if it was still inside the pipeline and not yet passed to sink, eager acknowl

Read avro files with wildcard

2024-07-18 Thread irakli.keshel...@sony.com
Hi all, I have a Flink application where I need to read in AVRO files from s3 which are partitioned by date and hour. I need to read in multiple dates, meaning I need to read files from multiple folders. Does anyone know how I can do this? My application is written in Scala using Flink 1.17.1.

Re: Event de duplication in flink with rabbitmq connector

2024-07-18 Thread banu priya
Hi All, Gentle reminder about bow query. Thanks Banu On Tue, 9 Jul, 2024, 1:42 pm banu priya, wrote: > Hi All, > > I have a Flink job with a RMQ source, tumbling windows (fires for each > 2s), an aggregator, then a RMQ sink. Incremental RocksDB checkpointing is > enabled with an interval of 5