Issue with Flink - unrelated executeSql causes other executeSqls to fail.

I'm writing a test for a batch job using MiniClusterResourceConfiguration. Here's a simple description of my working test case: 1) I use TableEnvironment.executeSql(...) to create a source and sink table using tmp filesystem directory. 2) I use executeSql to insert some test data into the source t

Re: flink checkpoint timeout

I'm not 100% sure but from the given information this might be related to FLINK-14498 [1] and partially relieved by FLINK-16645 [2]. @Omkar Could you try the 1.11.0 release out and see whether the issue disappeared? @zhijiang @yingjie could you also take a look here? Thanks. Best Regards, Yu [

what's the example for datastream data generator?

I want to simulate the scene"back pressure"could you tell me what's the example for datastream datagen? Thanks for your help.

Guide on writing Flink plugins

Hi all, We want to write a Flink plugins to integrate Flink jobs with our in-house monitoring system. Are there any guide or tutorial that we can follow to write a Flink plugins ? The official documents are a bit bare bone. Regards, Kien

Reading from Multiple Directories with StreamExecutionEnvironment

Hi all, I came across this change that allows user to have multiple file paths to read from in Flink. However, I have a question about how to use this feature for StreamExecutionEnvironment.readFile(). It seems in readFile, the input filePaths

Applying Custom metrics

Hi I have questions regarding making my own custom metrics. When exactly is the class RichMapFunction’s map(value) method called/invoked, and what “value” will be passed/expected as an argument to this map(value) method? Does the RichMapFunction’s map() method have any relation to the transforma

ConnectionPool to DB and parallelism of operator question

HI, Basic question on parallelism of operators and ConnectionPool to DB: Will this result in 82 * 300 connections to InfluxDB or just 300 connections to InfluxDB ? main() { sink = createInfluxMonitoringSink(..); keyStream.addSink(sink).addParallelism(82);//will this result in 82 * 300 conne

List of Running Flink Jobs from Long Running Flink Yarn Cluster (AWS EMR)

Hi, I am trying to create a monitor task which periodically checks how many Flink Jobs are running with in a given Flink Yarn Session. In EMR we are starting the Flink Yarn Session using the following Command in detached mode flink-yarn-session -n 2 -d And all the Flink jobs we are sub

Re: Transition Flink job from Java to Scala with state migration

Thanks for the pointers Andrey, they helped us zero in on the issue. We found that it was a bug in our Scala code that caused Flink to use Kryo serializer to read Avro-based state. We've fixed it, and are able to evolve state-schema seamlessly now. Cheers, Daksh On Tue, Aug 25, 2020 at 9:53 PM A

Re: Stateful Functions + ML model prediction

Thanks for the response Gordon, and that FlinkForward presentation - it's been very helpful. I put in a JIRA ticket for it: https://issues.apache.org/jira/browse/FLINK-19507 I did find this page: https://ci.apache.org/projects/flink/flink-statefun-docs-release-2.2/io-module/flink-connectors.htm

?????? need help about "incremental checkpoint",Thanks

Could you give more details? Thanks --  -- ??: "??"

Re: Flink SQL - can I force the planner to reuse a temporary view to be reused across multiple queries that access different fields?

Great that you could fix it. Thanks for letting us know. Regards, Timo On 02.10.20 21:05, Dan Hill wrote: Thanks, Timo and Piotr! I figured out my issue.  I called env.disableOperatorChaining(); in my developer mode.  Disabling operator chaining created the redundant joins. On Mon, Sep 2

Re: Streaming SQL Job Switches to FINISHED before all records processed

Btw which planner are you using? Regards, Timo On 05.10.20 10:23, Timo Walther wrote: Hi Austin, could you share some details of your SQL query with us? The reason why I'm asking is because I guess that the rowtime field is not inserted into the `StreamRecord` of DataStream API. The rowtime

Re: Streaming SQL Job Switches to FINISHED before all records processed

Hi Austin, could you share some details of your SQL query with us? The reason why I'm asking is because I guess that the rowtime field is not inserted into the `StreamRecord` of DataStream API. The rowtime field is only inserted if there is a single field in the output of the query that is a

Re: Streaming SQL Job Switches to FINISHED before all records processed

Hi Austin, thanks for offering to help. First I would suggest asking Timo whether this is an aspect which is still missing or whether we overlooked it. Based on that we can then take the next steps. Cheers, Till On Fri, Oct 2, 2020 at 7:05 PM Austin Cawley-Edwards < austin.caw...@gmail.com> wrot

Re: Stateful Functions + ML model prediction

Hi John, It is definitely possible to use Apache Pulsar with StateFun. Could you open a JIRA ticket for that? It would be nice to see how much interest we can gather on adding that as a new IO module, and consider adding native support for Pulsar in future releases. If you are already using State