Re:Found an issue when using Flink 1.19's AsyncScalarFunction

2024-04-09 Thread Xuyang
Hi, Wang. Could you provide more details for this bug, such as minimum reproducible test code, pom dependencies, etc? Further more, can you try again to package the dependency "commons-text" with version "1.10.0" manually to check if it works? If you can work around this bug by this way, I t

Re: Flink 1.18.1 cannot read from Kafka

2024-04-09 Thread Biao Geng
Hi Phil, Thanks for sharing the detailed information of the job. For you question, how to you submit the job? After applying your yaml file, I think you will successfully launch a flink cluster with 1 JM and 1 TM. Then you would submit the pyflink job to the flink cluster. As the error you showed

Flink 1.18.1 cannot read from Kafka

2024-04-09 Thread Phil Stavridis
Hello, I have set up Flink and Kafka containers using docker-compose, for testing how Flink works for processing Kafka messages. I primarily want to check how the Table API works but also how the Stream API would process the Kafka messages. I have included the main part of the docker-compose.ya

Re: Use of data generator source

2024-04-09 Thread Lasse Nedergaard
Hi TkachenkoYes I have and we use it extensively for unit testing. But we also have integration testing as part of our project and here I run into the problem.In my previous implementation I used SourceFunction interface and added a delay in the run function. but it’s depredicated so I have changed

Re: Use of data generator source

2024-04-09 Thread Yaroslav Tkachenko
Hi Lasse, Have you seen this approach https://nightlies.apache.org/flink/flink-docs-release-1.19/docs/dev/datastream/testing/#unit-testing-stateful-or-timely-udfs--custom-operators ? On Tue, Apr 9, 2024 at 7:09 AM Lasse Nedergaard < lassenedergaardfl...@gmail.com> wrote: > Hi. > > I my Integrati

Re: Debugging Kryo Fallback

2024-04-09 Thread Salva Alcántara
{ "emoji": "👍", "version": 1 }

Use of data generator source

2024-04-09 Thread Lasse Nedergaard
Hi. I my Integration test, running on 1.19, with a mini cluster I mock all my sources with DataGeneratorSource and it works fine until I have a timer function in a key processed function. The problem is that the processing time doesn’t advance after all data has been produced in the sources.

Re: Debugging Kryo Fallback

2024-04-09 Thread Zhanghao Chen
Hi, you may first enable the Kryo fallback option first, submit the job, and search for "be processed as GenericType". Flink will print it in most cases where we fall back to Kryo (a few exceptions including type Class, Object, recursive types, interface). Best, Zhanghao Chen __

Found an issue when using Flink 1.19's AsyncScalarFunction

2024-04-09 Thread Xiaolong Wang
Hi, I found a ClassNotFound exception when using Flink 1.19's AsyncScalarFunction. Stack trace: Caused by: java.lang.ClassNotFoundException: > org.apache.commons.text.StringSubstitutor > > at java.net.URLClassLoader.findClass(Unknown Source) ~[?:?] > > at java.lang.ClassLoader.loadClass(Unknown