Hi Pachara, Could you check if this is a similar issue of https://issues.apache.org/jira/browse/FLINK-37266. If so, you could try to replace `-pyclientexec` with configuration option `python.client.executable` to see if it works.
Regards, Dian On Mon, Jul 7, 2025 at 3:39 PM Nikola Milutinovic <n.milutino...@levi9.com> wrote: > > Hi Pachara. > > > > We are also struggling to get a PyFlink job to run. Hitting some other wall, > but I would like to share some thoughts here. > > > > At the moment, we are launching our fleet of jobs using a specially crafted > Docker image: Flink 1.20.1 + plugins + PyFlink + our Python libs + our jobs. > From that image we are invoking “flink run -py … -pyFile …”. So, that image > is running as a Flink client and it interacts with Flink Session Cluster. > > > > The reason why I am mentioning all of this is that – I suspect – under > covers, Flink K8s Operator works in similar fashion. When you make a K8s > Custom Resource, the Operator will act as a client and it will try to read > those files you mention. > > > > And there lies the problem. file:///flink-tmp/... Is a reference to a local > file, but local to what? Kubernetes operator, I suspect. That is why some > examples show the following: > > > > spec: > > job: > > jarURI: local:///opt/flink/lib/flink-python-1.20.1.jar > > … > > > > This fails for us, stating that “local” is not a recognized URI schema. Never > mind, we skipped it (and hit another wall). > > > > Sooo,…. You could try putting your Python files on some global storage, like > S3 and see if that works. If it does, please tell us. > > > > Nikola. > > > > From: Pachara Aryuyuen (Kas) <paryuy...@zilo.co.uk> > Date: Monday, July 7, 2025 at 8:38 AM > To: user@flink.apache.org <user@flink.apache.org> > Cc: Santhana Jinjakam (Eye) <sjinja...@zilo.co.uk> > Subject: [PyFlink] Issue Deploying FlinkSessionJob with PVC and Python Script > Access > > Dear Flink Community, > > I’m currently working on deploying a FlinkSessionJob using PyFlink and have > encountered an issue I hope someone can help clarify. > > Here’s my setup: > > I installed the Flink Kubernetes Operator in namespace A > I deployed a FlinkDeployment in namespace B > I’m trying to deploy a FlinkSessionJob in namespace B > > The job fails with a file not found error related to the jarURI. I’m > referencing a JAR file located at /flink-tmp/, which is mounted as a PVC on > the FlinkDeployment. This PVC is backed by EFS, so it should be accessible > across pods. I’ve confirmed that the file exists at that path. > > I also tried using the following public JAR instead: > > jarURI: > https://repo1.maven.org/maven2/org/apache/flink/flink-python/1.20.1/flink-python-1.20.1.jar > > However, even with this change, the Python script located in the same mounted > volume (/flink-tmp/transform_confirmed_trade.py) still cannot be found. > > I haven’t been able to find any best practices or working examples of > FlinkSessionJob with PyFlink. Do you have any sample manifests or guidance > for this use case? > > Here is the manifest I’m using: > > apiVersion: flink.apache.org/v1beta1 > > kind: FlinkSessionJob > > metadata: > > name: transform-confirmed-trade-job > > namespace: namespace-b > > spec: > > deploymentName: flink-cluster > > job: > > jarURI: file:///flink-tmp/flink-python-1.20.1.jar > > entryClass: "org.apache.flink.client.python.PythonDriver" > > args: ["-pyclientexec", "/usr/bin/python3", "--python", > "/flink-tmp/transform_confirmed_trade.py"] > > parallelism: 2 > > upgradeMode: stateless > > Additional Question: > > Is it possible to configure TaskManagers to run permanently (i.e., not shut > down when idle)? If so, what configuration options are required to achieve > this behavior? > > Any help or pointers would be greatly appreciated. > > Best regards, > > Pachara Aryuyuen (Kas) > > Cloud Platform Engineer > >
