Hi Matt, Yes. There are several official Flink images with various JVMs including Java 11.
https://hub.docker.com/_/flink Cheers, Matyas On Fri, Jun 24, 2022 at 2:06 PM Matt Casters <matt.cast...@neotechnology.com> wrote: > Hi Mátyás & all, > > Thanks again for the advice so far. On a related note I noticed Java 8 > being used, indicated in the log. > > org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - > JAVA_HOME: /usr/local/openjdk-8 > > Is there a way to use Java 11 to start Flink with? > > Kind regards, > > Matt > > On Tue, Jun 21, 2022 at 4:53 PM Őrhidi Mátyás <matyas.orh...@gmail.com> > wrote: > >> Hi Matt, >> >> I believe an artifact fetcher (e.g >> https://hub.docker.com/r/agiledigital/s3-artifact-fetcher ) + the pod >> template ( >> https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/custom-resource/pod-template/#pod-template) >> is an elegant way to solve your problem. >> >> The operator uses K8s native integration under the hood: >> https://nightlies.apache.org/flink/flink-docs-release-1.13/docs/deployment/resource-providers/native_kubernetes/#application-mode >> In >> application mode, the main() method of the application is executed on the >> JobManager, hence we need the jar locally. >> >> You can launch a session cluster (without job spec) on the operator that >> allows submitting jars if you would like to avoid dealing with >> authentication, but the recommended and safe approach is to use >> sessionjobs for this purpose. >> >> >> Cheers, >> Matyas >> >> On Tue, Jun 21, 2022 at 4:03 PM Matt Casters < >> matt.cast...@neotechnology.com> wrote: >> >>> Thank you very much for the help Matyas and Gyula! >>> >>> I just saw a video today where you were presenting the FKO. Really nice >>> stuff! >>> >>> So I'm guessing we're executing "flink run" at some point on the master >>> and that this is when we need the jar file to be local? >>> Am I right in assuming that this happens after the flink cluster in >>> question was started, as part of the job execution? >>> >>> On the one hand I agree with the underlying idea that authentication and >>> security should not be a responsibility of the operator. On the other >>> hand I could add a flink-s3 driver but then I'd also have to configure it >>> and so on and it's just hard to get that configuration to be really clean. >>> >>> Do we have some service running on the flink cluster which would allow >>> us to post/copy files from the client (running kubectl) to the master? If >>> so, could we add an option to the job specification to that effect? Just >>> brainstorming ;-) (and forking apache/flink-kubernetes-operator) >>> >>> All the best, >>> Matt >>> >>> On Tue, Jun 21, 2022 at 2:52 PM Őrhidi Mátyás <matyas.orh...@gmail.com> >>> wrote: >>> >>>> Hi Matt, >>>> >>>> - In FlinkDeployments you can utilize an init container to download >>>> your artifact onto a shared volume, then you can refer to it as local:/.. >>>> from the main container. FlinkDeployments comes with pod template support >>>> https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/custom-resource/pod-template/#pod-template >>>> >>>> - FlinkSessionJobs comes with an artifact fetcher, but it may need some >>>> tweaking to make it work on your environment: >>>> >>>> https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/custom-resource/overview/#flinksessionjob-spec-overview >>>> >>>> I hope it helps, let us know if you have further questions. >>>> >>>> Cheers, >>>> Matyas >>>> >>>> >>>> >>>> On Tue, Jun 21, 2022 at 2:35 PM Matt Casters < >>>> matt.cast...@neotechnology.com> wrote: >>>> >>>>> Hi Flink team! >>>>> >>>>> I'm interested in getting the new Flink Kubernetes Operator to work on >>>>> AWS EKS. Following the documentation I got pretty far. However, when >>>>> trying to run a job I got the following error: >>>>> >>>>> Only "local" is supported as schema for application mode. This assumes >>>>>> t >>>>>> hat the jar is located in the image, not the Flink client. An example >>>>>> of such path is: local:///opt/flink/examples/streaming/WindowJoin.jar >>>>> >>>>> >>>>> I have an Apache Hop/Beam fat jar capable of running the Flink >>>>> pipeline in my yml file: >>>>> >>>>> jarURI: s3://hop-eks/hop/hop-2.1.0-fat.jar >>>>> >>>>> So how could I go about getting the fat jar in a desired location for >>>>> the operator? >>>>> >>>>> Getting this to work would be really cool for both short and >>>>> long-lived pipelines in the service of all sorts of data integration work. >>>>> It would do away with the complexity of setting up and maintaining your >>>>> own >>>>> Flink cluster. >>>>> >>>>> Thanks in advance! >>>>> >>>>> All the best, >>>>> >>>>> Matt (mcasters, Apache Hop PMC) >>>>> >>>>>
