Hi guys,
I m very close to deploying my application in production so I am trying to
externalize some of the config files which has to be available on the
classpath when I run my application via flink command line interface.

>From the flink doc I can add to class path by

-C,--classpath <url>                 Adds a URL to each user code
                                          classloader  on all nodes in the
                                          cluster. The paths must specify a
                                          protocol (e.g. file://) and be
                                          accessible on all nodes (e.g. by means
                                          of a NFS share). You can use this
                                          option multiple times for specifying
                                          more than one URL. The protocol must
                                          be supported by the {@link
                                          java.net.URLClassLoader}.

So my job has 8 task managers( 8 different nodes) with 8 slots in each .
Does my external class path have to be on NFS share ? Can I not have the
config directory on each machine in the same location ? For instance on
Node 1 the config files are in  the directory is
/home/was/classpathconfig/  and
the same on every node. Does it have to be on an NFS ?
My command looks like this flink run -C file://home/was/classpathconfig/ -c
com.visa.flink.cli.Main flink-job-assembly-0.1-SNAPSHOT.jar flink druid

Also I tried to put my files in s3 and tried to run flink run -C s3://
flink.dev/config -c com.visa.flink.cli.Main
flink-job-assembly-0.1-SNAPSHOT.jar flink druid

Bad syntax for classpath: s3://flink.dev/config


s3 does support URLClassLoader but I get the error saying bad syntax.


Please let me know your thoughts. Thanks a lot to this community , I
was able to write my code in a week.


Thanks,

Vishwas

Reply via email to