Hi Vishwas, > Does my external class path have to be on NFS share ? Can I not have the config directory on each machine in the same location ?
It can be local files on each machine. > -C file://home/was/classpathconfig/ I guess the problem is your URI format of local file is not correct. Try to use "-C file:///home/was/classpathconfig/" instead. BTW, if it's a directory, it must end with '/' [1]. > Also I tried to put my files in s3 and tried to run flink run -C s3:// flink.dev/config -c com.visa.flink.cli.Main flink-job-assembly-0.1-SNAPSHOT.jar flink druid Currently it doesn't support s3 or other distributed file system as the parameter of -C. 1. https://docs.oracle.com/javase/8/docs/api/java/net/URLClassLoader.html Thanks, Biao /'bɪ.aʊ/ On Thu, 15 Aug 2019 at 07:51, Vishwas Siravara <vsirav...@gmail.com> wrote: > Hi guys, > I m very close to deploying my application in production so I am trying to > externalize some of the config files which has to be available on the > classpath when I run my application via flink command line interface. > > From the flink doc I can add to class path by > > -C,--classpath <url> Adds a URL to each user code > classloader on all nodes in the > cluster. The paths must specify a > protocol (e.g. file://) and be > accessible on all nodes (e.g. by > means > of a NFS share). You can use this > option multiple times for specifying > more than one URL. The protocol must > be supported by the {@link > java.net.URLClassLoader}. > > So my job has 8 task managers( 8 different nodes) with 8 slots in each . > Does my external class path have to be on NFS share ? Can I not have the > config directory on each machine in the same location ? For instance on > Node 1 the config files are in the directory is > /home/was/classpathconfig/ and the same on every node. Does it have to > be on an NFS ? > My command looks like this flink run -C file://home/was/classpathconfig/ > -c com.visa.flink.cli.Main flink-job-assembly-0.1-SNAPSHOT.jar flink druid > > Also I tried to put my files in s3 and tried to run flink run -C s3:// > flink.dev/config -c com.visa.flink.cli.Main > flink-job-assembly-0.1-SNAPSHOT.jar flink druid > > Bad syntax for classpath: s3://flink.dev/config > > > s3 does support URLClassLoader but I get the error saying bad syntax. > > > Please let me know your thoughts. Thanks a lot to this community , I was able > to write my code in a week. > > > Thanks, > > Vishwas > >
