Hello Deniz, Looking at /flink/usrlib and the way it is expected to be used, Flink will only pick up .jar files and include them into the classpath, so unfortunately it is being excluded. If you want it to just make it work and get on with your day, you can simply place module.yaml in a separate JAR , otherwise keep on reading :-)
I've created a branch[1], that supports providing a custom name for the module.yaml, if you are comfortable with building this branch and trying it out, I can go forward with adding this to statefun, as I believe others might need a similar functionality. To make it work you need: 1. Use @Fabian Paul <fabian.paul...@gmail.com>'s advice and upload your custom module.yaml as you did before, you can also rename it now to whatever name you want. For example prod.yaml. 2. This file will appear at /flink/usrlib/prod.yaml 3. You would also need to specify this file by adding the following to your flink-conf.yaml: statefun.remote.module-name: /flink/usrlib/prod.yaml 4. At the bottom of this page [2] you can see a full example, and how to add additional flink configurations (flinkConfiguration) property. I hope this helps, Igal. [1] https://github.com/igalshilman/flink-statefun/tree/custom_module [2] https://docs.ververica.com/user_guide/application_operations/deployments/index.html#deployment-defaults On Thu, Dec 9, 2021 at 12:22 PM Deniz Koçak <lend...@gmail.com> wrote: > Hi Fabian, > > Thanks for that solution.. I've removed the module.yaml file from the > jar file assuming that it should be fetched from s3 and used by the > job. I've tried this on our job, but still its seems to be failing. > > From the logs module.yaml file seems to be fetched from s3 bucket. > ---- > com.ververica.platform.artifactfetcher.ArtifactFetcherEntryPoint - > Finished fetching > s3://rttk8s-nxt-v2/vvp/artifacts/namespaces/default/module.yaml into > /flink/usrlib/module.yaml > ---- > > However we got that exception below: > > ---- > Caused by: java.lang.IllegalStateException: There are no ingress > defined. at > org.apache.flink.statefun.flink.core.StatefulFunctionsUniverseValidator.validate(StatefulFunctionsUniverseValidator.java:25) > ~[?:?] > ---- > > Please let me know if you need further information. Thanks again for your > help. > > Deniz > > On Wed, Dec 8, 2021 at 1:20 PM Fabian Paul <fp...@apache.org> wrote: > > > > Hi Deniz, > > > > Great to hear from someone using Ververica Platform with StateFun. > > When deploying your job you can specify `additionalConfigurations`[1] > > that are also pulled and put into the classpath. > > > > Hopefully, that is suitable for your scenario. > > > > Best, > > Fabian > > > > [1] > https://docs.ververica.com/user_guide/application_operations/deployments/artifacts.html?highlight=additionaldependencies > > > > On Fri, Dec 3, 2021 at 4:51 PM Deniz Koçak <lend...@gmail.com> wrote: > > > > > > Hi Igal, > > > > > > We are using official images from Ververica as the Flink installation. > > > Actually, I was hoping to specify the name of file names to use during > > > the runtime via `mainArgs` in the deployment configuration (or any > > > other way may be). By this way we can specify the target yaml files, > > > but I think this is not possible? > > > > > > ======================= > > > kind: JAR > > > mainArgs: '--active-profile nxt' > > > ======================= > > > > > > Therefore, it's easier to use single jar in our pipelines instead of > > > creating a different jar file for each env. (at least for development > > > and production). > > > > > > For solution 2, you refer flink distro. , like /flink/lib folder in > > > the official Docker image? > > > > > > Thanks, > > > Deniz > > > > > > On Fri, Dec 3, 2021 at 3:06 PM Igal Shilman <i...@apache.org> wrote: > > > > > > > > Hi Deniz, > > > > > > > > StateFun would be looking for module.yaml(s) in the classpath. > > > > If you are submitting the job to an existing Flink cluster this > really means that it needs to be either: > > > > 1. packaged with the jar (like you are already doing) > > > > 2. be present at the classpath, this means that you can place your > module.yaml at the /lib directory of your Flink installation, I suppose > that you have different installations in different environments. > > > > > > > > I'm not aware of a way to submit any additional files with the jar > via the flink cli, but perhaps someone else can chime in :-) > > > > > > > > Cheers, > > > > Igal. > > > > > > > > > > > > On Thu, Dec 2, 2021 at 3:29 PM Deniz Koçak <lend...@gmail.com> > wrote: > > > >> > > > >> Hi, > > > >> > > > >> We have a simple stateful-function job, consuming from Kafka, > calling > > > >> an HTTP endpoint (on AWS via an Elastic Load Balancer) and > publishing > > > >> the result back via Kafka again. > > > >> > > > >> * We created a jar file to be deployed on a standalone cluster (it's > > > >> not a docker Image), therefore we add `statefun-flink-distribution` > > > >> version 3.0.0 as a dependency in that jar file. > > > >> * Entry class in our job configuration is > > > >> `org.apache.flink.statefun.flink.core.StatefulFunctionsJob` and we > > > >> simply keep a single module.yaml file in resources folder for the > > > >> module configuration. > > > >> > > > >> My question here is, we would like to deploy that jar to different > > > >> environments (dev. and prod.) and not sure how we can pass different > > > >> module configurations (module.yaml or > module_nxt.yaml/module_prd.yaml) > > > >> to the job during startup without creating separate jar files for > > > >> different environments? > > > >> > > > >> Thanks, > > > >> Deniz >
