What I mean is you could create a K8s deployment using the Flink image just
like following.
After then, you could use "kubectl exec -it {pod_name} bash" to tunnel in
and submit the Flink
python application to the existing session cluster.
apiVersion: apps/v1
kind: Deployment
metadata:
name: flink-client
spec:
replicas: 1
selector:
matchLabels:
app: flink-client
template:
metadata:
labels:
app: flink-client
spec:
containers:
- name: client
image: flink:1.12.2
imagePullPolicy: Always
args: ["sleep", "86400"]
Best,
Yang
Robert Cullen <cinquate...@gmail.com> 于2021年3月10日周三 下午9:36写道：
> Yang,
>
> "... This could be done by starting a pod as the Flink client."
>
> Do you have an example yaml you could share?
>
> On Sun, Mar 7, 2021 at 9:57 PM Yang Wang <danrtsey...@gmail.com> wrote:
>
>> I think you want to submit a Flink python job to the existing session
>> cluster.
>> Please ensure the session cluster is created with proper service exposed
>> type[1].
>> * LoadBalancer for the cloud environment
>> * NodePort for self managed K8s cluster
>> * ClusterIP for the K8s internal submission, which means you should
>> submit the Flink application in the K8s cluster. This could be done by
>> starting a pod as the Flink client.
>>
>> [1].
>> https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/resource-providers/native_kubernetes/#accessing-flinks-web-ui
>>
>> Best,
>> Yang
>>
>> Robert Cullen <cinquate...@gmail.com> 于2021年3月6日周六 上午12:43写道：
>>
>>> Trying to spin up a Python Flink instance in my Kubernetes cluster with
>>> this configuration ...
>>>
>>> sudo ./bin/flink run \
>>> --target kubernetes-session \
>>> -Dkubernetes.cluster-id=flink-python \
>>> -Dkubernetes.namespace=cmdaa \
>>> -Dkubernetes.container.image=cmdaa/pyflink:0.0.1 \
>>> --pyModule word_count \
>>> --pyFiles /opt/flink-1.12.2/examples/python/table/batch/word_count.py
>>>
>>> ... But getting this error:
>>>
>>> Traceback (most recent call last):
>>> File "/Users/admin/.pyenv/versions/3.6.10/lib/python3.6/runpy.py", line
>>> 193, in _run_module_as_main
>>> "__main__", mod_spec)
>>> File "/Users/admin/.pyenv/versions/3.6.10/lib/python3.6/runpy.py", line
>>> 85, in _run_code
>>> exec(code, run_globals)
>>> File
>>> "/var/folders/zz/zyxvpxvq6csfxvn_n0000000000000/T/pyflink/074c581c-5b39-4b07-ac84-55a86c46f9eb/affe3559-5364-43f1-93ef-b7f6cd9a2f77/word_count.py",
>>> line 80, in <module>
>>> word_count()
>>> File
>>> "/var/folders/zz/zyxvpxvq6csfxvn_n0000000000000/T/pyflink/074c581c-5b39-4b07-ac84-55a86c46f9eb/affe3559-5364-43f1-93ef-b7f6cd9a2f77/word_count.py",
>>> line 74, in word_count
>>> t_env.execute("word_count")
>>> File
>>> "/opt/flink-1.12.2/opt/python/pyflink.zip/pyflink/table/table_environment.py",
>>> line 1276, in execute
>>> File
>>> "/opt/flink-1.12.2/opt/python/py4j-0.10.8.1-src.zip/py4j/java_gateway.py",
>>> line 1286, in __call__
>>> File
>>> "/opt/flink-1.12.2/opt/python/pyflink.zip/pyflink/util/exceptions.py", line
>>> 147, in deco
>>> File
>>> "/opt/flink-1.12.2/opt/python/py4j-0.10.8.1-src.zip/py4j/protocol.py", line
>>> 328, in get_return_value
>>> py4j.protocol.Py4JJavaError: An error occurred while calling o2.execute.
>>> : java.lang.RuntimeException:
>>> org.apache.flink.client.deployment.ClusterRetrieveException: Could not get
>>> the rest endpoint of flink-python
>>> at
>>> org.apache.flink.kubernetes.KubernetesClusterDescriptor.lambda$createClusterClientProvider$0(KubernetesClusterDescriptor.java:102)
>>> at
>>> org.apache.flink.kubernetes.KubernetesClusterDescriptor.retrieve(KubernetesClusterDescriptor.java:145)
>>> at
>>> org.apache.flink.kubernetes.KubernetesClusterDescriptor.retrieve(KubernetesClusterDescriptor.java:66)
>>> at
>>> org.apache.flink.client.deployment.executors.AbstractSessionClusterExecutor.execute(AbstractSessionClusterExecutor.java:75)
>>> at
>>> org.apache.flink.table.api.internal.BatchTableEnvImpl.executePipeline(BatchTableEnvImpl.scala:347)
>>> at
>>> org.apache.flink.table.api.internal.BatchTableEnvImpl.execute(BatchTableEnvImpl.scala:317)
>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> at
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>> at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>> at java.lang.reflect.Method.invoke(Method.java:498)
>>> at
>>> org.apache.flink.api.python.shaded.py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
>>> at
>>> org.apache.flink.api.python.shaded.py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
>>> at
>>> org.apache.flink.api.python.shaded.py4j.Gateway.invoke(Gateway.java:282)
>>> at
>>> org.apache.flink.api.python.shaded.py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
>>> at
>>> org.apache.flink.api.python.shaded.py4j.commands.CallCommand.execute(CallCommand.java:79)
>>> at
>>> org.apache.flink.api.python.shaded.py4j.GatewayConnection.run(GatewayConnection.java:238)
>>> at java.lang.Thread.run(Thread.java:748)
>>> Caused by: org.apache.flink.client.deployment.ClusterRetrieveException:
>>> Could not get the rest endpoint of flink-python
>>> ... 17 more
>>>
>>> org.apache.flink.client.program.ProgramAbortException
>>> at
>>> org.apache.flink.client.python.PythonDriver.main(PythonDriver.java:124)
>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> at
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>> at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>> at java.lang.reflect.Method.invoke(Method.java:498)
>>> at
>>> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:349)
>>> at
>>> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:219)
>>> at
>>> org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:114)
>>> at
>>> org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:812)
>>> at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:246)
>>> at
>>> org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:1054)
>>> at
>>> org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:1132)
>>> at
>>> org.apache.flink.runtime.security.contexts.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:28)
>>> at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1132)
>>>
>>> --
>>> Robert Cullen
>>> 240-475-4490
>>>
>>
>
> --
> Robert Cullen
> 240-475-4490
>
