Glad to hear that. And the repository you shared is very helpful for the users who are trying to deploy Flink clusters on K8s.
Best, Yang Felipe Gutierrez <felipe.o.gutier...@gmail.com> 于2020年9月24日周四 下午2:04写道： > thanks Yang, > I got to put it to work in the way that you said. > https://github.com/felipegutierrez/explore-flink > > Best, > Felipe > -- > -- Felipe Gutierrez > -- skype: felipe.o.gutierrez > -- https://felipeogutierrez.blogspot.com > > On Thu, Sep 24, 2020 at 6:59 AM Yang Wang <danrtsey...@gmail.com> wrote: > > > > Hi Felipe, > > > > Currently, if you want to deploy a standalone job/application Flink > cluster on K8s via yamls. > > You should have your own image with user jar baked located at > /opt/flink/usrlib. It could not be > > specified via config option. Usually, you could add new layer on the > official docker image to > > build in the user jar. > > > > A more graceful solution is init container[1], which could download the > user jar from remote > > storage or just copy from local directory /opt/flink/opt to > /opt/flink/usrlib. > > > > [1]. > https://kubernetes.io/docs/tasks/configure-pod-container/configure-pod-initialization/ > > > > Best, > > Yang > > > > > > Felipe Gutierrez <felipe.o.gutier...@gmail.com> 于2020年9月18日周五 下午5:52写道： > >> > >> Hi community, > >> > >> I am trying to deploy the default WordCount stream application on > >> minikube using the official documentation at [1]. I am using minikube > >> v1.13.0 on Ubuntu 18.04 and Kubernetes v1.19.0 on Docker 19.03.8. I > >> could sucessfully start 1 job manager and 3 task managers using the > >> yaml files flink-configuration-configmap.yaml, > >> jobmanager-service.yaml, jobmanager-rest-service.yaml, > >> jobmanager-session-deployment.yaml, and > >> taskmanager-session-deployment.yaml (all available on the Apendix of > >> this link [1]). > >> > >> Then I would like to start the word-count stream job available on the > >> flink jar image [2], which I believe is available since it is built > >> inside the default flink jar distribution. What I understood that I > >> have to do is to create the objects based on the files > >> jobmanager-job.yaml and taskmanager-job-deployment.yaml (also > >> available on the link [1]). And, I think that I have to replace this > >> line below on the object jobmanager-job.yaml > >> (spec.template.spec.containers.name[jobmanager]): > >> > >> args: ["standalone-job", "--job-classname", > >> "org.apache.flink.streaming.examples.wordcount.WordCount"] > >> > >> Is this correct? I am not sure if this is my entire error. I am > >> getting the message "Could not find the provided job class > >> (org.apache.flink.streaming.examples.wordcount.WordCount) in the user > >> lib directory (/opt/flink/usrlib)". As far as I know > >> "/opt/flink/usrlib" is the default directory. I am not sure if I have > >> to change the property: path /host/path/to/job/artifacts. This is my > >> log message of the pod error. > >> Do you guys have any idea of what I am missing in my configuration? > >> > >> Thanks, Felipe > >> > >> $ kubectl get pods > >> NAME READY STATUS RESTARTS AGE > >> flink-jobmanager-ftgg9 0/1 CrashLoopBackOff 3 83s > >> $ kubectl logs flink-jobmanager-ftgg9 > >> Starting Job Manager > >> sed: couldn't open temporary file /opt/flink/conf/sedA699Jt: Read-only > >> file system > >> sed: couldn't open temporary file /opt/flink/conf/sedvZhs0w: Read-only > >> file system > >> /docker-entrypoint.sh: 72: /docker-entrypoint.sh: cannot create > >> /opt/flink/conf/flink-conf.yaml: Permission denied > >> /docker-entrypoint.sh: 91: /docker-entrypoint.sh: cannot create > >> /opt/flink/conf/flink-conf.yaml.tmp: Read-only file system > >> Starting standalonejob as a console application on host > flink-jobmanager-ftgg9. > >> 2020-09-18 09:15:58,801 INFO > >> org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - > >> > -------------------------------------------------------------------------------- > >> 2020-09-18 09:15:58,804 INFO > >> org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - > >> Preconfiguration: > >> 2020-09-18 09:15:58,804 INFO > >> org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - > >> > >> > >> JM_RESOURCE_PARAMS extraction logs: > >> jvm_params: -Xmx1073741824 -Xms1073741824 -XX:MaxMetaspaceSize=268435456 > >> logs: INFO [] - Loading configuration property: > >> jobmanager.rpc.address, flink-jobmanager > >> INFO [] - Loading configuration property: > taskmanager.numberOfTaskSlots, 4 > >> INFO [] - Loading configuration property: blob.server.port, 6124 > >> INFO [] - Loading configuration property: jobmanager.rpc.port, 6123 > >> INFO [] - Loading configuration property: taskmanager.rpc.port, 6122 > >> INFO [] - Loading configuration property: queryable-state.proxy.ports, > 6125 > >> INFO [] - Loading configuration property: > jobmanager.memory.process.size, 1600m > >> INFO [] - Loading configuration property: > >> taskmanager.memory.process.size, 1728m > >> INFO [] - Loading configuration property: parallelism.default, 2 > >> INFO [] - The derived from fraction jvm overhead memory (160.000mb > >> (167772162 bytes)) is less than its min value 192.000mb (201326592 > >> bytes), min value will be used instead > >> INFO [] - Final Master Memory configuration: > >> INFO [] - Total Process Memory: 1.563gb (1677721600 bytes) > >> INFO [] - Total Flink Memory: 1.125gb (1207959552 bytes) > >> INFO [] - JVM Heap: 1024.000mb (1073741824 bytes) > >> INFO [] - Off-heap: 128.000mb (134217728 bytes) > >> INFO [] - JVM Metaspace: 256.000mb (268435456 bytes) > >> INFO [] - JVM Overhead: 192.000mb (201326592 bytes) > >> > >> 2020-09-18 09:15:58,805 INFO > >> org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - > >> > -------------------------------------------------------------------------------- > >> 2020-09-18 09:15:58,805 INFO > >> org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - > >> Starting StandaloneApplicationClusterEntryPoint (Version: 1.11.0, > >> Scala: 2.11, Rev:d04872d, Date:2020-06-29T16:13:14+02:00) > >> 2020-09-18 09:15:58,805 INFO > >> org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - OS > >> current user: flink > >> 2020-09-18 09:15:58,805 INFO > >> org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - > >> Current Hadoop/Kerberos user: <no hadoop dependency found> > >> 2020-09-18 09:15:58,805 INFO > >> org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - > >> JVM: OpenJDK 64-Bit Server VM - Oracle Corporation - 1.8/25.262-b10 > >> 2020-09-18 09:15:58,806 INFO > >> org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - > >> Maximum heap size: 989 MiBytes > >> 2020-09-18 09:15:58,806 INFO > >> org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - > >> JAVA_HOME: /usr/local/openjdk-8 > >> 2020-09-18 09:15:58,806 INFO > >> org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - No > >> Hadoop Dependency available > >> 2020-09-18 09:15:58,806 INFO > >> org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - JVM > >> Options: > >> 2020-09-18 09:15:58,806 INFO > >> org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - > >> -Xmx1073741824 > >> 2020-09-18 09:15:58,806 INFO > >> org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - > >> -Xms1073741824 > >> 2020-09-18 09:15:58,807 INFO > >> org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - > >> -XX:MaxMetaspaceSize=268435456 > >> 2020-09-18 09:15:58,807 INFO > >> org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - > >> > -Dlog.file=/opt/flink/log/flink--standalonejob-0-flink-jobmanager-ftgg9.log > >> 2020-09-18 09:15:58,807 INFO > >> org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - > >> -Dlog4j.configuration=file:/opt/flink/conf/log4j-console.properties > >> 2020-09-18 09:15:58,807 INFO > >> org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - > >> -Dlog4j.configurationFile=file:/opt/flink/conf/log4j-console.properties > >> 2020-09-18 09:15:58,807 INFO > >> org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - > >> -Dlogback.configurationFile=file:/opt/flink/conf/logback-console.xml > >> 2020-09-18 09:15:58,807 INFO > >> org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - > >> Program Arguments: > >> 2020-09-18 09:15:58,807 INFO > >> org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - > >> --configDir > >> 2020-09-18 09:15:58,808 INFO > >> org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - > >> /opt/flink/conf > >> 2020-09-18 09:15:58,808 INFO > >> org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - > >> --job-classname > >> 2020-09-18 09:15:58,808 INFO > >> org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - > >> org.apache.flink.streaming.examples.wordcount.WordCount > >> 2020-09-18 09:15:58,808 INFO > >> org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - > >> Classpath: > /opt/flink/lib/flink-csv-1.11.0.jar:/opt/flink/lib/flink-json-1.11.0.jar:/opt/flink/lib/flink-shaded-zookeeper-3.4.14.jar:/opt/flink/lib/flink-table-blink_2.11-1.11.0.jar:/opt/flink/lib/flink-table_2.11-1.11.0.jar:/opt/flink/lib/log4j-1.2-api-2.12.1.jar:/opt/flink/lib/log4j-api-2.12.1.jar:/opt/flink/lib/log4j-core-2.12.1.jar:/opt/flink/lib/log4j-slf4j-impl-2.12.1.jar:/opt/flink/lib/flink-dist_2.11-1.11.0.jar::: > >> 2020-09-18 09:15:58,808 INFO > >> org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - > >> > -------------------------------------------------------------------------------- > >> 2020-09-18 09:15:58,809 INFO > >> org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - > >> Registered UNIX signal handlers for [TERM, HUP, INT] > >> 2020-09-18 09:15:58,849 ERROR > >> org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - > >> Could not create application program. > >> org.apache.flink.util.FlinkException: Could not find the provided job > >> class (org.apache.flink.streaming.examples.wordcount.WordCount) in the > >> user lib directory (/opt/flink/usrlib). > >> at > org.apache.flink.client.deployment.application.ClassPathPackagedProgramRetriever.getJobClassNameOrScanClassPath(ClassPathPackagedProgramRetriever.java:140) > >> ~[flink-dist_2.11-1.11.0.jar:1.11.0] > >> at > org.apache.flink.client.deployment.application.ClassPathPackagedProgramRetriever.getPackagedProgram(ClassPathPackagedProgramRetriever.java:123) > >> ~[flink-dist_2.11-1.11.0.jar:1.11.0] > >> at > org.apache.flink.container.entrypoint.StandaloneApplicationClusterEntryPoint.getPackagedProgram(StandaloneApplicationClusterEntryPoint.java:110) > >> ~[flink-dist_2.11-1.11.0.jar:1.11.0] > >> at > org.apache.flink.container.entrypoint.StandaloneApplicationClusterEntryPoint.main(StandaloneApplicationClusterEntryPoint.java:78) > >> [flink-dist_2.11-1.11.0.jar:1.11.0] > >> > >> [1] > https://ci.apache.org/projects/flink/flink-docs-stable/ops/deployment/kubernetes.html > >> [2] > https://github.com/apache/flink/blob/master/flink-examples/flink-examples-streaming/src/main/java/org/apache/flink/streaming/examples/wordcount/WordCount.java > >> -- > >> -- Felipe Gutierrez > >> -- skype: felipe.o.gutierrez > >> -- https://felipeogutierrez.blogspot.com >
