Re: Using pyflink from flink distribution

Great, thank you so much for your responses. It all makes sense now. :) On Mon, Jan 30, 2023 at 10:41 PM Dian Fu wrote: > >> What is the reason for including > opt/python/{pyflink.zip,cloudpickle.zip,py4j.zip} in the base > distribution then? Oh, a guess: to make it easier for TaskManagers to

Re: Using pyflink from flink distribution

>> What is the reason for including opt/python/{pyflink.zip,cloudpickle.zip,py4j.zip} in the base distribution then? Oh, a guess: to make it easier for TaskManagers to run pyflink without having pyflink installed themselves? Somehow I'd guess this wouldn't work tho; I'd assume TaskManagers would

Re: Using pyflink from flink distribution

Thanks Dian! > >> Is using pyflink from the flink distribution tarball (without pip) not a supported way to use pyflink? > You are right. What is the reason for including opt/python/{pyflink.zip,cloudpickle.zip,py4j.zip} in the base distribution then? Oh, a guess: to make it easier for TaskManag

Re: Using pyflink from flink distribution

Hi Andrew, >> By pip installing apache-flink, this docker image will have the flink distro installed at /opt/flink and FLINK_HOME set to /opt/flink . BUT ALSO flink lib jars will be installed at e.g. /usr/

Re: Using pyflink from flink distribution

Ah, oops and my original email had a typo: > Some python dependencies are not included in the flink distribution tarballs: cloudpickle, py4j and pyflink are in opt/python. Should read: > Some python dependencies ARE included in the flink distribution tarballs: cloudpickle, py4j and pyflink are in

Re: Using pyflink from flink distribution

Let me ask a related question: We are building our own base Flink docker image. We will be deploying both JVM and python apps via flink-kubernetes-operator. Is there any reason not to install Flink in this image via `pip install apache-flink` and use it for JVM apps? -Andrew Otto Wikimedia Fou