Re: Guide for building Flink image with Python doesn't work

Here it is, copied from the docs essentially: FROM flink:1.15.0 # install python3: it has updated Python to 3.9 in Debian 11 and so install Python 3.7 from source # it currently only supports Python 3.6, 3.7 and 3.8 in PyFlink officially. RUN apt-get update -y && \ apt-get install -y build-esse

Re: ContinuousFileMonitoringFunction retrieved invalid state.

Hi Vishal, The FLINK-28274 is on going. And I think a better solution is that you can migrate your job to the new FileSource[1], because this issue only occurs when using the legacy file source(which will be deprecated soon). [1] https://nightl

Re: how to connect to the flink-state store and use it as cache to serve APIs.

Hi Folks, I just wanted to double check, if there is any way to expose rest APIs using Flink sql tables ? Thank you. On Thu, Jun 30, 2022 at 12:15 PM Yuan Mei wrote: > That's definitely something we want to achieve in the future term, and > your input is very valuable. > > One problem with

Re: Guide for building Flink image with Python doesn't work

Hi Gyula, I can build pyFlink image successfully by following this guide. Did you add a dependency outside of the documentation? And could you provide your Dockerfile https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/deployment/resource-providers/standalone/docker/#using-flink-pytho

Re: How can I convert a DataSet into a Table?

What's the version of Flink you are using? In Flink 1.13, you can use BatchTableEnvironment#fromDataSet() to do that. But since Flink 1.14, the method has been removed. [1] https://nightlies.apache.org/flink/flink-docs-release-1.13/api/java/org/apache/flink/table/api/bridge/java/BatchTableEnvi

How can I convert a DataSet into a Table?

  My code is:   package flinkTest2;   import org.apache.flink.api.java.DataSet; import org.apache.flink.api.java.ExecutionEnvironment; import org.apache.flink.api.java.tuple.Tuple2; import org.apache.flink.table.api.EnvironmentSettings; import org.apache.flink.table.api.Table; import org.apac

Re: Best practice for creating/restoring savepoint in standalone k8 setup

Hi Jonas! I think generally managed platforms used to provide the functionality that you are after. Otherwise it's mostly home grown CI/CD integrations :) The Kubernetes Operator is maybe the first initiative to bring proper application lifecycle management to the ecosystem directly. Cheers, Gyu

Re: Best practice for creating/restoring savepoint in standalone k8 setup

Thanks Weihua and Gyula, @Weihia > If you restart flink cluster by delete/create deployment directly, it will be automatically restored from the latest checkpoint[1], so maybe just enabling the checkpoint is enough. Not sure I follow, we might have changes to the job that will require us to restor

Re: Guide for building Flink image with Python doesn't work

Well in any case either the official image is incorrect (maybe we should include JDK by default not JRE) or we should update the documentation regarding the python docker build because it simply doesn't work at the moment. I am still looking for a full working example that adds the required Python

Re: Best practice for creating/restoring savepoint in standalone k8 setup

Hi! Did you check the https://github.com/apache/flink-kubernetes-operator by any chance? It provides many of the application lifecycle features that you are probably after straight out-of-the-box. It has both manual and periodic savepoint trig

Re: Guide for building Flink image with Python doesn't work

In addition, you can try providing the Dockerfile Best, Weihua On Tue, Jul 5, 2022 at 11:24 PM Weihua Hu wrote: > Hi, > > The base image flink:1.15.0 is built from openjdk:11-jre, and this image > only installs jre but not jdk. > It looks like the package you want to install (pemja) depends on

Re: Best practice for creating/restoring savepoint in standalone k8 setup

Hi, jonas If you restart flink cluster by delete/create deployment directly, it will be automatically restored from the latest checkpoint[1], so maybe just enabling the checkpoint is enough. But if you want to use savepoint, you need to check whether the latest savepoint is successful (check wheth

Re: Guide for building Flink image with Python doesn't work

Hi, The base image flink:1.15.0 is built from openjdk:11-jre, and this image only installs jre but not jdk. It looks like the package you want to install (pemja) depends on jdk. you need install openjdk-11-jdk in dockerfile, take a look to how it is installed in the official image: https://hub.do

Best practice for creating/restoring savepoint in standalone k8 setup

Hi! We are running a Standalone job on Kubernetes using application deployment mode, with HA enabled. We have attempted to automate how we create and restore savepoints by running a script for generating a savepoint (using k8 preStop hook) and another one for restoring from a savepoint (located i

Re: Recover watermark from savepoint

Hi Thias, Thank you for providing a detailed explanation. We did something similar. The job is set to 0 late events, aggregates every 20 min and sends out the value. So we are saving the last processed window per key in the state. During aggregation, if current window > last window, then processe

Guide for building Flink image with Python doesn't work

Hi All! I have been trying to experiment with the Flink python support on Kubernetes but I got stuck creating a custom image with all the necessary python libraries. I found this guide in the docs: https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/deployment/resource-providers/stand