Re: FlinkKinesisConsumer: Dropping records on deserialiuation problems

2022-08-03 Thread Martijn Visser
Hi Peter, It was definitely not a bad idea! Good luck and if you need more help, feel free to reach out to the mailing list once more. Best regards, Martijn Op wo 3 aug. 2022 om 16:24 schreef Peter Schrott : > Hi Martjin, > > Thanks a lot for that example. This looks a lot like my suggested >

Re: FlinkKinesisConsumer: Dropping records on deserialiuation problems

2022-08-03 Thread Peter Schrott
Hi Martjin, Thanks a lot for that example. This looks a lot like my suggested workaround. Good to know that my idea was not too naive. All the best, Peter > On 3. Aug 2022, at 16:16, Martijn Visser wrote: > > Hi Peter, > > You could consider the pattern that was used to create a Kafka Dead L

Re: FlinkKinesisConsumer: Dropping records on deserialiuation problems

2022-08-03 Thread Martijn Visser
Hi Peter, You could consider the pattern that was used to create a Kafka Dead Letter Queue. There's a recipe including source code available for that at https://docs.immerok.cloud/docs/cookbook/creating-dead-letter-queues-from-and-to-apache-kafka-with-apache-flink/ Best regards, Martijn Op wo

FlinkKinesisConsumer: Dropping records on deserialiuation problems

2022-08-03 Thread Peter Schrott
Hi Flink Ppl! Working with Apache Flink v 1.13.2 on AWS with Kinesis as source. I have the requirement to drop certain events before they enter the jobs pipeline. There are mainly 2 reasons: 1) Problems when it comes to deserializing the incoming bytes 2) Problems with the event itself, e.g. mis