Re: Best practice for packaging and deploying Flink jobs on K8S

2021-05-02 Thread Sumeet Malhotra
Thanks for updating the documentation Dian. Appreciate it. ..Sumeet On Sun, May 2, 2021 at 10:53 AM Dian Fu wrote: > Hi Sumeet, > > FYI: the documentation about the CLI options of PyFlink has already been > updated [1]. > > [1] > https://ci.apache.org/projects/flink/flink-docs-release-1.12/depl

Re: Best practice for packaging and deploying Flink jobs on K8S

2021-05-01 Thread Dian Fu
Hi Sumeet, FYI: the documentation about the CLI options of PyFlink has already been updated [1]. [1] https://ci.apache.org/projects/flink/flink-docs-release-1.12/deployment/cli.html#submitting-pyflink-jobs Regards, Dian On Thu, Apr 29, 2021 at 4:46 PM Dian Fu wrote: > Hi Sumeet, > > For the

Re: Best practice for packaging and deploying Flink jobs on K8S

2021-04-29 Thread Dian Fu
Hi Sumeet, For the Python dependencies, multiple ways have been provided to specify them and you could take either way of them. Regarding to requirements.txt, there are 3 ways provided and you could specify it via either of them: - API inside the code: set_python_requirements - command line opt

Re: Best practice for packaging and deploying Flink jobs on K8S

2021-04-29 Thread Till Rohrmann
Alright. Then let's see what Dian recommends to do. Cheers, Till On Thu, Apr 29, 2021 at 9:25 AM Sumeet Malhotra wrote: > Hi Till, > > There’s no problem with the documented approach. I was looking if there > were any standardized ways of organizing, packaging and deploying Python > code on a F

Re: Best practice for packaging and deploying Flink jobs on K8S

2021-04-29 Thread Sumeet Malhotra
Hi Till, There’s no problem with the documented approach. I was looking if there were any standardized ways of organizing, packaging and deploying Python code on a Flink cluster. Thanks, Sumeet On Thu, Apr 29, 2021 at 12:37 PM Till Rohrmann wrote: > Hi Sumeet, > > Is there a problem with the

Re: Best practice for packaging and deploying Flink jobs on K8S

2021-04-29 Thread Till Rohrmann
Hi Sumeet, Is there a problem with the documented approaches on how to submit the Python program (not working) or are you asking in general? Given the documentation, I would assume that you can configure the requirements.txt via `set_python_requirements`. I am also pulling in Dian who might be ab

Best practice for packaging and deploying Flink jobs on K8S

2021-04-28 Thread Sumeet Malhotra
Hi, I have a PyFlink job that consists of: - Multiple Python files. - Multiple 3rdparty Python dependencies, specified in a `requirements.txt` file. - A few Java dependencies, mainly for external connectors. - An overall job config YAML file. Here's a simplified structure of the c