sjwiesman commented on a change in pull request #11878: URL: https://github.com/apache/flink/pull/11878#discussion_r417611803
########## File path: docs/dev/table/python/usage_notes.md ########## @@ -0,0 +1,120 @@ +--- +title: "Usage Notes" +nav-parent_id: python_tableapi +nav-pos: 140 +--- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +This page shows solutions to common problems encountered by PyFlink users after getting started. + +* This will be replaced by the TOC +{:toc} +## How to Add Jars +A PyFlink job often depends on some jar packages, such as connector jar packages, java udf jar packages and so on. +So how to make PyFlink refer to these jar packages? + +You can specify the dependencies with the following Python Table APIs or through command line arguments directly when submitting the job。 Review comment: ```suggestion ## How to Add Jars A PyFlink job often depends on some jar packages, such as connector jar packages, java UDF jar packages, and so on. So how to make PyFlink refer to these jar packages? You can specify the dependencies with the following Python Table APIs or through command-line arguments directly when submitting the job. ``` ########## File path: docs/dev/table/python/usage_notes.md ########## @@ -0,0 +1,120 @@ +--- +title: "Usage Notes" Review comment: What do you think about changing this to "Common Questions"? ```suggestion title: "Common Questions" ``` ########## File path: docs/dev/table/python/usage_notes.md ########## @@ -0,0 +1,120 @@ +--- +title: "Usage Notes" +nav-parent_id: python_tableapi +nav-pos: 140 +--- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +This page shows solutions to common problems encountered by PyFlink users after getting started. + +* This will be replaced by the TOC +{:toc} +## How to Add Jars +A PyFlink job often depends on some jar packages, such as connector jar packages, java udf jar packages and so on. +So how to make PyFlink refer to these jar packages? + +You can specify the dependencies with the following Python Table APIs or through command line arguments directly when submitting the job。 +{% highlight python %} +table_env.get_config().set_configuration("pipeline.jars", "file:///my/jar/path/connector.jar;file:///my/jar/path/udf.jar") +{% endhighlight %} +For details, you can refer to <a href="{{ site.baseurl }}/dev/table/python/dependency_management.html#usage">dependency management + +## How to Watch UDF Log +There are different solutions for watching UDF log in local mode and cluster mode. +#### Local +you will see all udf log infos in the console directly. Review comment: ```suggestion When running locally, all you will see all UDF logs in the console directly. ``` ########## File path: docs/dev/table/python/usage_notes.md ########## @@ -0,0 +1,120 @@ +--- +title: "Usage Notes" +nav-parent_id: python_tableapi +nav-pos: 140 +--- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +This page shows solutions to common problems encountered by PyFlink users after getting started. + +* This will be replaced by the TOC +{:toc} +## How to Add Jars +A PyFlink job often depends on some jar packages, such as connector jar packages, java udf jar packages and so on. +So how to make PyFlink refer to these jar packages? + +You can specify the dependencies with the following Python Table APIs or through command line arguments directly when submitting the job。 +{% highlight python %} +table_env.get_config().set_configuration("pipeline.jars", "file:///my/jar/path/connector.jar;file:///my/jar/path/udf.jar") +{% endhighlight %} +For details, you can refer to <a href="{{ site.baseurl }}/dev/table/python/dependency_management.html#usage">dependency management + +## How to Watch UDF Log +There are different solutions for watching UDF log in local mode and cluster mode. +#### Local +you will see all udf log infos in the console directly. +#### Cluster +you can see the udf log infos in the taskmanager log. Review comment: ```suggestion On cluster deployments, you can see the UDF logs in the taskmanager logs. ``` ########## File path: docs/dev/table/python/usage_notes.md ########## @@ -0,0 +1,120 @@ +--- +title: "Usage Notes" +nav-parent_id: python_tableapi +nav-pos: 140 +--- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +This page shows solutions to common problems encountered by PyFlink users after getting started. + +* This will be replaced by the TOC +{:toc} +## How to Add Jars +A PyFlink job often depends on some jar packages, such as connector jar packages, java udf jar packages and so on. +So how to make PyFlink refer to these jar packages? + +You can specify the dependencies with the following Python Table APIs or through command line arguments directly when submitting the job。 +{% highlight python %} +table_env.get_config().set_configuration("pipeline.jars", "file:///my/jar/path/connector.jar;file:///my/jar/path/udf.jar") +{% endhighlight %} +For details, you can refer to <a href="{{ site.baseurl }}/dev/table/python/dependency_management.html#usage">dependency management + +## How to Watch UDF Log +There are different solutions for watching UDF log in local mode and cluster mode. +#### Local +you will see all udf log infos in the console directly. +#### Cluster +you can see the udf log infos in the taskmanager log. + +## How to Prepare a Python Virtual Env Used by PyFlink +You can refer to the following script to prepare a Python virtual env zip which can be used in mac os and most Linux distributions Review comment: Instead of this, why don't we include the below script as a downloadable bash script? You can simply copy the below into a file `setup-pyflink-virtual-env.sh` in the same directory and then the sentence could be the following: ```suggestion You can download a [convenience script](setup-pyflink-virtual-env.sh) to prepare a Python virtual env zip which can be used on Mac OS and most Linux distirbutions. ``` ########## File path: docs/dev/table/python/usage_notes.md ########## @@ -0,0 +1,120 @@ +--- +title: "Usage Notes" +nav-parent_id: python_tableapi +nav-pos: 140 +--- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +This page shows solutions to common problems encountered by PyFlink users after getting started. + +* This will be replaced by the TOC +{:toc} +## How to Add Jars +A PyFlink job often depends on some jar packages, such as connector jar packages, java udf jar packages and so on. +So how to make PyFlink refer to these jar packages? + +You can specify the dependencies with the following Python Table APIs or through command line arguments directly when submitting the job。 +{% highlight python %} +table_env.get_config().set_configuration("pipeline.jars", "file:///my/jar/path/connector.jar;file:///my/jar/path/udf.jar") +{% endhighlight %} +For details, you can refer to <a href="{{ site.baseurl }}/dev/table/python/dependency_management.html#usage">dependency management + +## How to Watch UDF Log +There are different solutions for watching UDF log in local mode and cluster mode. +#### Local +you will see all udf log infos in the console directly. +#### Cluster +you can see the udf log infos in the taskmanager log. + +## How to Prepare a Python Virtual Env Used by PyFlink +You can refer to the following script to prepare a Python virtual env zip which can be used in mac os and most Linux distributions +{% highlight shell %} +# you need to install wget and zip manually +echo "required command: wget, zip" + +# download miniconda.sh +set -e +if [ `uname -s` == "Darwin" ]; then + # If you live in China, you can change the url to https://mirrors.tuna.tsinghua.edu.cn/anaconda/miniconda/Miniconda3-4.7.10-MacOSX-x86_64.sh + wget "https://repo.continuum.io/miniconda/Miniconda3-4.7.10-MacOSX-x86_64.sh"; -O "miniconda.sh" +else + # If you live in China, you can change the url to https://mirrors.tuna.tsinghua.edu.cn/anaconda/miniconda/Miniconda3-4.7.10-Linux-x86_64.sh + wget "https://repo.continuum.io/miniconda/Miniconda3-4.7.10-Linux-x86_64.sh"; -O "miniconda.sh" +fi + +chmod +x miniconda.sh + +# create python virtual env +./miniconda.sh -b -p venv +TMP_PATH=$PATH +PATH=`cd venv/bin;pwd`:$PATH + +# install dependency +pip install apache-flink +rm -rf venv/pkgs +PATH=$TMP_PATH +zip -r venv.zip venv +{% endhighlight %} + +## How to Run Python UDF Locally +Firstly, you need to prepare a Python Virtual Env Used by PyFlink (You can refer to the previous section). + +Then, you should execute the following script to activate your used Python virtual environment Review comment: ```suggestion To run Python UDFs locally, first, prepare a Python Virtual Env used by PyFlink as described in the previous section. Then, execute the following script to activate your environment. ``` ########## File path: docs/dev/table/python/usage_notes.md ########## @@ -0,0 +1,120 @@ +--- +title: "Usage Notes" +nav-parent_id: python_tableapi +nav-pos: 140 +--- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +This page shows solutions to common problems encountered by PyFlink users after getting started. + +* This will be replaced by the TOC +{:toc} +## How to Add Jars +A PyFlink job often depends on some jar packages, such as connector jar packages, java udf jar packages and so on. +So how to make PyFlink refer to these jar packages? + +You can specify the dependencies with the following Python Table APIs or through command line arguments directly when submitting the job。 +{% highlight python %} +table_env.get_config().set_configuration("pipeline.jars", "file:///my/jar/path/connector.jar;file:///my/jar/path/udf.jar") +{% endhighlight %} +For details, you can refer to <a href="{{ site.baseurl }}/dev/table/python/dependency_management.html#usage">dependency management + +## How to Watch UDF Log +There are different solutions for watching UDF log in local mode and cluster mode. +#### Local +you will see all udf log infos in the console directly. +#### Cluster +you can see the udf log infos in the taskmanager log. + +## How to Prepare a Python Virtual Env Used by PyFlink +You can refer to the following script to prepare a Python virtual env zip which can be used in mac os and most Linux distributions +{% highlight shell %} +# you need to install wget and zip manually +echo "required command: wget, zip" + +# download miniconda.sh +set -e +if [ `uname -s` == "Darwin" ]; then + # If you live in China, you can change the url to https://mirrors.tuna.tsinghua.edu.cn/anaconda/miniconda/Miniconda3-4.7.10-MacOSX-x86_64.sh + wget "https://repo.continuum.io/miniconda/Miniconda3-4.7.10-MacOSX-x86_64.sh"; -O "miniconda.sh" +else + # If you live in China, you can change the url to https://mirrors.tuna.tsinghua.edu.cn/anaconda/miniconda/Miniconda3-4.7.10-Linux-x86_64.sh + wget "https://repo.continuum.io/miniconda/Miniconda3-4.7.10-Linux-x86_64.sh"; -O "miniconda.sh" +fi + +chmod +x miniconda.sh + +# create python virtual env +./miniconda.sh -b -p venv +TMP_PATH=$PATH +PATH=`cd venv/bin;pwd`:$PATH + +# install dependency +pip install apache-flink +rm -rf venv/pkgs +PATH=$TMP_PATH +zip -r venv.zip venv +{% endhighlight %} + +## How to Run Python UDF Locally +Firstly, you need to prepare a Python Virtual Env Used by PyFlink (You can refer to the previous section). + +Then, you should execute the following script to activate your used Python virtual environment + +{% highlight shell %} +$ venv/bin/conda init +# you need to restart the shell +$ conda activate +{% endhighlight %} +Now, you can run your Python UDF script directly. +{% highlight shell %} +python xxx.py +{% endhighlight %} + +## How to Run Python UDF in Cluster +There are two ways of preparing a Python environment installed PyFlink which can be used in the Cluster Review comment: ```suggestion There are two ways of preparing a Python environment installed PyFlink, which can be used in the Cluster. ``` ########## File path: docs/dev/table/python/usage_notes.md ########## @@ -0,0 +1,120 @@ +--- +title: "Usage Notes" +nav-parent_id: python_tableapi +nav-pos: 140 +--- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +This page shows solutions to common problems encountered by PyFlink users after getting started. + +* This will be replaced by the TOC +{:toc} +## How to Add Jars +A PyFlink job often depends on some jar packages, such as connector jar packages, java udf jar packages and so on. +So how to make PyFlink refer to these jar packages? + +You can specify the dependencies with the following Python Table APIs or through command line arguments directly when submitting the job。 +{% highlight python %} +table_env.get_config().set_configuration("pipeline.jars", "file:///my/jar/path/connector.jar;file:///my/jar/path/udf.jar") +{% endhighlight %} +For details, you can refer to <a href="{{ site.baseurl }}/dev/table/python/dependency_management.html#usage">dependency management + +## How to Watch UDF Log +There are different solutions for watching UDF log in local mode and cluster mode. +#### Local +you will see all udf log infos in the console directly. +#### Cluster +you can see the udf log infos in the taskmanager log. + +## How to Prepare a Python Virtual Env Used by PyFlink +You can refer to the following script to prepare a Python virtual env zip which can be used in mac os and most Linux distributions +{% highlight shell %} +# you need to install wget and zip manually +echo "required command: wget, zip" + +# download miniconda.sh +set -e +if [ `uname -s` == "Darwin" ]; then + # If you live in China, you can change the url to https://mirrors.tuna.tsinghua.edu.cn/anaconda/miniconda/Miniconda3-4.7.10-MacOSX-x86_64.sh + wget "https://repo.continuum.io/miniconda/Miniconda3-4.7.10-MacOSX-x86_64.sh"; -O "miniconda.sh" +else + # If you live in China, you can change the url to https://mirrors.tuna.tsinghua.edu.cn/anaconda/miniconda/Miniconda3-4.7.10-Linux-x86_64.sh + wget "https://repo.continuum.io/miniconda/Miniconda3-4.7.10-Linux-x86_64.sh"; -O "miniconda.sh" +fi + +chmod +x miniconda.sh + +# create python virtual env +./miniconda.sh -b -p venv +TMP_PATH=$PATH +PATH=`cd venv/bin;pwd`:$PATH + +# install dependency +pip install apache-flink +rm -rf venv/pkgs +PATH=$TMP_PATH +zip -r venv.zip venv +{% endhighlight %} + +## How to Run Python UDF Locally +Firstly, you need to prepare a Python Virtual Env Used by PyFlink (You can refer to the previous section). + +Then, you should execute the following script to activate your used Python virtual environment + +{% highlight shell %} +$ venv/bin/conda init +# you need to restart the shell +$ conda activate +{% endhighlight %} +Now, you can run your Python UDF script directly. +{% highlight shell %} +python xxx.py +{% endhighlight %} + +## How to Run Python UDF in Cluster +There are two ways of preparing a Python environment installed PyFlink which can be used in the Cluster + +#### Install Packages in the machine of the cluster + +{% highlight shell %} +$ pip install apache-flink +{% endhighlight %} + +After installing, you can use `pyexec` command line arg or `set_python_executable` api of table_env to specify the path of the python interpreter which is used to execute the python udf workers. + +#### Use Uploaded Virtual Environment +You can use `pyarch` command line arg or `add_python_archive` api of table_env to upload Python virtual environment and use `pyexec` command line arg or `set_python_executable` api to specify the path of the python interpreter which is used to execute the python udf workers. + +For details about the command line args of `pyarch` and `pyexec`, you can refer to <a href="{{ site.baseurl }}/ops/cli.html#usage">command line arguments.</a> Review comment: ```suggestion For details about the command line args of `pyarch` and `pyexec`, you can refer to [the relevant documentation]({{ site.baseurl }}/ops/cli.html#usage). ``` ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
