Hi guys,
I m very close to deploying my application in production so I am trying to
externalize some of the config files which has to be available on the
classpath when I run my application via flink command line interface.
>From the flink doc I can add to class path by
-C,--classpath <url> Adds a URL to each user code
classloader on all nodes in the
cluster. The paths must specify a
protocol (e.g. file://) and be
accessible on all nodes (e.g. by means
of a NFS share). You can use this
option multiple times for specifying
more than one URL. The protocol must
be supported by the {@link
java.net.URLClassLoader}.
So my job has 8 task managers( 8 different nodes) with 8 slots in each .
Does my external class path have to be on NFS share ? Can I not have the
config directory on each machine in the same location ? For instance on
Node 1 the config files are in the directory is
/home/was/classpathconfig/ and
the same on every node. Does it have to be on an NFS ?
My command looks like this flink run -C file://home/was/classpathconfig/ -c
com.visa.flink.cli.Main flink-job-assembly-0.1-SNAPSHOT.jar flink druid
Also I tried to put my files in s3 and tried to run flink run -C s3://
flink.dev/config -c com.visa.flink.cli.Main
flink-job-assembly-0.1-SNAPSHOT.jar flink druid
Bad syntax for classpath: s3://flink.dev/config
s3 does support URLClassLoader but I get the error saying bad syntax.
Please let me know your thoughts. Thanks a lot to this community , I
was able to write my code in a week.
Thanks,
Vishwas
