[ANNOUNCE] Share your Streaming Stories with us at Current 2023

2023-01-24 Thread Israel Ekpo
Do you have a great data streaming story to share? We want to hear from you! Speaking at Current 2023 is a great way to connect with hundreds of your peers, become more involved in the data streaming community, and have a public platform for you to share your story of the future of streaming and

Using pyflink from flink distribution

2023-01-24 Thread Andrew Otto
Hello, I'm having quite a bit of trouble running pyflink from the default flink distribution tarballs. I'd expect the python examples to work as long as python is installed, and we've got the distribution. Some python dependencies are not included in the flink distribution tarballs: cloudpickle,

Failed Checkpoints when Flink HPA enabled on Kubernetes

2023-01-24 Thread Varun Narayanan Chakravarthy via user
Hello Flink Users, We have enabled Kubernetes HPA for our Flink applications (Task Manager only). Our Flink Applications are running in Reactive Mode. When the Kubernetes controller triggers a scale-up/scale-down of our job's Task Managers, we get an alert for failed checkpoints. Interestingly enou

Re: Blob server connection problem

2023-01-24 Thread Matthias Pohl via user
We had issues like that in the past (e.g. FLINK-24923 [1], FLINK-10683 [2]). The error you're observing is caused by an unexpected byte being read from the socket. The BlobServer protocol expects either 0 (for put messages) or 1 (for get messages) being retrieved as a header for new message blocks

Re: Problem with custom SerializationSchema in Flink 1.15

2023-01-24 Thread Chesnay Schepler
It's a known issue that various connectors/wrappers/etc did not respect the schema lifecycle. This was fixed in 1.16.0 in https://issues.apache.org/jira/browse/FLINK-28807. You will have to lazily initialize the mapper in the serialize() method for previous versions. On 24/01/2023 11:52, P

Problem with custom SerializationSchema in Flink 1.15

2023-01-24 Thread Peter Schrott
Hi Flink-User! I recently updated a Flink job from Flink version 1.13 to 1.15 (managed by AWS). The Flink Job is written in Java. I found out that the Kinesis Producer was deprecated in favour of Kinesis Streams Sink [1]. When upgrading to the new sink I stumbled upon a problem withe a custom

Using S3 as stream source in Flink

2023-01-24 Thread Sriram Ganesh
Hi Everyone, I am thinking of switching my input source from Kafka to S3. First, I couldn't find any streaming source connector for S3. I have some basic questions about 1. How S3 will work as a streaming source with proper checkpointing.? 2. How Flink will manage the last offset processed from a