Hi guys, I m very close to deploying my application in production so I am trying to externalize some of the config files which has to be available on the classpath when I run my application via flink command line interface.
>From the flink doc I can add to class path by -C,--classpath <url> Adds a URL to each user code classloader on all nodes in the cluster. The paths must specify a protocol (e.g. file://) and be accessible on all nodes (e.g. by means of a NFS share). You can use this option multiple times for specifying more than one URL. The protocol must be supported by the {@link java.net.URLClassLoader}. So my job has 8 task managers( 8 different nodes) with 8 slots in each . Does my external class path have to be on NFS share ? Can I not have the config directory on each machine in the same location ? For instance on Node 1 the config files are in the directory is /home/was/classpathconfig/ and the same on every node. Does it have to be on an NFS ? My command looks like this flink run -C file://home/was/classpathconfig/ -c com.visa.flink.cli.Main flink-job-assembly-0.1-SNAPSHOT.jar flink druid Also I tried to put my files in s3 and tried to run flink run -C s3:// flink.dev/config -c com.visa.flink.cli.Main flink-job-assembly-0.1-SNAPSHOT.jar flink druid Bad syntax for classpath: s3://flink.dev/config s3 does support URLClassLoader but I get the error saying bad syntax. Please let me know your thoughts. Thanks a lot to this community , I was able to write my code in a week. Thanks, Vishwas