Re: Flink task lifecycle listener/hook/SPI

2022-08-03 Thread Lijie Wang
Hi Allen, >From my experience, you can do your init setup by the following 2 ways: 1. Do your init setup in RichFunction#open method, see [1] for details. 2. Do your init setup in static block, it will be executed when the class is loaded. [1] https://nightlies.apache.org/flink/flink-docs-master/

RE: Flink support on Avro1.11

2022-08-03 Thread Deepti Sharma S via user
Hello Martijn, Thank you for the details. Just one more question when we have plan for Flink 1.16 release with Avro 1.11.0? Regards, Deepti Sharma PMP® & ITIL From: Martijn Visser Sent: Wednesday, August 3, 2022 6:50 PM To: Deepti Sharma S Cc: user Subject: Re: Flink support on Avro1.11

Re: Why this example does not save anything to file?

2022-08-03 Thread podunk
Thank you very much Martijn you dedicated your productive time to help me! I'm new noob in this subject - I took that example somewhere from Internet. I see problem for guys like me is that Flink syntax changes from version to version quite significantly. So here not 'connector.type' but 'conne

Re: FlinkKinesisConsumer: Dropping records on deserialiuation problems

2022-08-03 Thread Martijn Visser
Hi Peter, It was definitely not a bad idea! Good luck and if you need more help, feel free to reach out to the mailing list once more. Best regards, Martijn Op wo 3 aug. 2022 om 16:24 schreef Peter Schrott : > Hi Martjin, > > Thanks a lot for that example. This looks a lot like my suggested >

Re: FlinkKinesisConsumer: Dropping records on deserialiuation problems

2022-08-03 Thread Peter Schrott
Hi Martjin, Thanks a lot for that example. This looks a lot like my suggested workaround. Good to know that my idea was not too naive. All the best, Peter > On 3. Aug 2022, at 16:16, Martijn Visser wrote: > > Hi Peter, > > You could consider the pattern that was used to create a Kafka Dead L

Re: FlinkKinesisConsumer: Dropping records on deserialiuation problems

2022-08-03 Thread Martijn Visser
Hi Peter, You could consider the pattern that was used to create a Kafka Dead Letter Queue. There's a recipe including source code available for that at https://docs.immerok.cloud/docs/cookbook/creating-dead-letter-queues-from-and-to-apache-kafka-with-apache-flink/ Best regards, Martijn Op wo

Re: Why this example does not save anything to file?

2022-08-03 Thread Martijn Visser
I've verified your code locally and it doesn't work indeed, at least not with the latest Flink version (I've tested it with Flink 1.15). There are a couple of reasons for that: 1. You've mentioned in this thread that there's no problem with the 'csv.field-delimiter'. There is actually, because the

FlinkKinesisConsumer: Dropping records on deserialiuation problems

2022-08-03 Thread Peter Schrott
Hi Flink Ppl! Working with Apache Flink v 1.13.2 on AWS with Kinesis as source. I have the requirement to drop certain events before they enter the jobs pipeline. There are mainly 2 reasons: 1) Problems when it comes to deserializing the incoming bytes 2) Problems with the event itself, e.g. mis

Re: Flink support on Avro1.11

2022-08-03 Thread Martijn Visser
Hi Deepti Sharma, This was pending on the release of AVRO 1.11.1, due to a regression introduced in 1.11.0. Since that's just been released a couple of days ago, I've asked the original contributor if he can update his PR. You can track the Jira at https://issues.apache.org/jira/browse/FLINK-24718

Flink support on Avro1.11

2022-08-03 Thread Deepti Sharma S via user
Hello Team, Please confirm, if any Flink version supports Avro version 1.11? If not, what are the plans to support Avro v1.11? Regards, Deepti Sharma PMP(r) & ITIL

Pojo state schema evolution not working correctly

2022-08-03 Thread tao xiao
Hi team, I encountered below exception after I added a new field to a POJO used in list state and resumed the job from checkpoint > [error occurred during error reporting , id > 0xb]\n","stream":"stdout","time": > \n","stream":"stdout","time": > #\n","stream":"stdout","time": > # http://bugre

Re: Flink Operator - delete flinkdeployments

2022-08-03 Thread Gyula Fóra
Hi! We haven't experienced this problem in general (and other users also haven't reported). Could you please share your operator logs when you triggered the deletion that takes long? Long job cancellation times can lead to delayed deletion times. Cheers, Gyula On Wed, Aug 3, 2022 at 11:07 AM Si

Re: Flink Operator - delete flinkdeployments

2022-08-03 Thread Sigalit Eliazov
hello we upgraded to version 1.1.0 and i am afraid the problem exists in that version as well. I would appreciate any additional ideas or guidelines on how to do the cleanup correctly. thanks Sigalit On Tue, Aug 2, 2022 at 3:39 PM Sigalit Eliazov wrote: > Will do, thanks! > > On Tue, Aug 2, 2