[ANNOUNCE] Apache Flink 1.14.3 released

The Apache Flink community is very happy to announce the release of Apache Flink 1.14.3, which is the second bugfix release for the Apache Flink 1.14 series. Apache Flink® is an open-source stream processing framework for distributed, high-performing, always-available, and accurate data streaming

Re: Flink 1.11 Flink 1.13 savepoint compatiblity

Upon searching more, I found the Compatibility Table which does mention that Flink 1.11 savepoint is compatible with Flink 1.13 On Wed, Jan 19, 2022 at 1:34 PM Abhishek SP wrote: > Hello, > > Are Flink

Flink 1.11 Flink 1.13 savepoint compatiblity

Hello, Are Flink 1.11 savepoints compatible with Flink 1.13? We are trying to migrate a production Flink 1.11 app to Flink 1.13. The app uses RocksDB as the statebackend. This documentation

Re: [statefun] upgrade path - shared cluster use

Hello Fil, There is a PR opened today for upgrading the main branch to use the latest Flink version, and after its merged we'll kick the 3.2 release. Cheers, Igal On Mon 17. Jan 2022 at 13:53, Dawid Wysakowicz wrote: > I am pretty confident the goal is to be able to run on the newest Flink > ve

Re: Setting egress topic name in stateful functions module definition

Hi Deniz, I assume that you are using the Python SDK, Here is an example how to set the egress topic name[1] I hope that helps, Igal. [1] https://github.com/apache/flink-statefun-playground/blob/release-3.1/python/greeter/functions.py#L58 On Wed 19. Jan 2022 at 09:53, Deniz Koçak wrote: > Hi,

Re: Hive Source - SplitEnumeratorContext and its callAsync method - possible bug

Ok, I think it was premature alert :) 1. We have a framework guarantee that start method will be called only once per SplitEnumerator instance, hence context.callAsync will be called only once 2. callAsync uses ScheduledExecutorService::scheduleAtFixedRate under the hood so If any execution of thi

Hive Source - SplitEnumeratorContext and its callAsync method - possible bug

Hi, in documentation for SplitEnumeratorContext::callAsync method we read that: "(...) When this method is invoked multiple times, The Callables may be executed in a thread pool concurrently. It is important to make sure that the callable does not modify any shared state, especially the states th

build.gradle troubles with IntelliJ

Hi For my first project I followed : https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/dev/datastream/project-configuration/ where there a build.gradle example I created a directory, entered into it and did gradle init. The I created the build.gradle as per the example. I opened th

Re: Unhandled exception in flink 1.14.2

This is a serialization bug in Flink, see https://issues.apache.org/jira/browse/FLINK-24550. It will be fixed in the upcoming 1.14.3 release. On 19/01/2022 09:01, Caizhi Weng wrote: Hi! To print out gc logs of job manager you can add this configuration to flink-conf.yaml env.java.opts.jobma

FileSource Usage

Hello, We are using FileSource to process Parquet Files and had a few doubts around it. Would really appreciate if somebody can help answer them: 1. For a given file, does FileSource read the contents inside it in order ? In o

Setting egress topic name in stateful functions module definition

Hi, In an Kafka module definition [1], I want to explicitly set the name of the outbound Kafka topic in the egress definition below, but could not find any reference to it in the docs. We are not using Java SDK, which seems to be allowing us to set such details, so not sure how it could be handled

Re: Examples / Documentation for Flink ML 2

Hi Bonino, Definitely, it will be great to build up the Flink ML docs together based on your experience. Thanks! Dong On Wed, Jan 19, 2022 at 4:32 PM Bonino Dario wrote: > Hi Dong, > > Thank you for the reply. Since we are actually experimenting with the > Flink ML libraries, If you think it's

Re: Examples / Documentation for Flink ML 2

Hi Dong, Thank you for the reply. Since we are actually experimenting with the Flink ML libraries, If you think it's worth, we may contribute some documentation, e.g., tutorial based on what we learn while setting up our test project with Flink ML. Is it something that might be of interest fo

Re: Flink Kinesis connector - EFO connection error with http proxy settings

Hello Saravanan, Yes you are correct. EFO uses AWS SDK v2 and the builder does not set proxy configuration [1]. The polling (non EFO) mechanism is using AWS SDK v1 which has a more general configuration deserialiser, and hence proxy is configurable. I do not believe there is a workaround for this

Distributing Data to multiple kafka partitions while having a single flink instance or worker or slot

Hi,    Problem:            - Currently i am using flink as an embedded library in one of my application, eventually the application will be the Job and will be deployed in the flink cluster , but right its not a cluster but a standalone single process running flink within the same process.     

Re: Unhandled exception in flink 1.14.2

Hi! To print out gc logs of job manager you can add this configuration to flink-conf.yaml env.java.opts.jobmanager: -Xloggc:/tmp/jobmanager-gc.log -XX:+PrintGCDetails -XX:+PrintGCDateStamps This will print gc logs to /tmp/jobmanager-gc.log. I'm not familiar with the garbage collection metrics p

Re: JDBC read DB causeOutOfMemoryError: Java heap space

Should I change the query? something like below to add a limit? If no limit, does that mean flink will read whole huge table to memory and fetch and return 20 records each time? val query = String.format("SELECT * FROM %s limit 1000", tableName) On Tue, Jan 18, 2022 at 11:56 PM Qihua Yang wrote