Thanks Jacky Lau for starting this discussion. I understand that you are trying to find a convenient way to specify dependency jars along with user jar. However, let's try to narrow down by differentiating deployment modes.
# Standalone mode No matter you are using the standalone mode on virtual machine, or in a Kubernetes cluster, it is not very difficult to preparing user jar and all the dependencies under the $FLINK_HOME/usrlib directory. After then, they will be loaded by user classloader automatically. # Yarn We already have "--ship/-Dyarn.ship-files" to ship the dependency jars. # Native K8s Currently, only the local user jar in the image could be supported. And users could not specify dependency jars. A feasible solution is using the init-container(via pod template[1]) to download the user jar and dependencies and then mount to usrlib directory. All in all, I trying to get you point about why do we need the "--jars" to specify the dependency jars. And which deployment mode it will support? Best, Yang [1]. https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/deployment/resource-providers/native_kubernetes/#pod-template Martijn Visser <martijnvis...@apache.org> 于2022年10月27日周四 14:49写道： > Hi Jacky Lau, > > Since you've sent the email to multiple mailing lists, I've decided to > reply to the one that you've sent to both the Dev and User ML. > > > but it is not possible for platform users to create fat jars to package > all their dependencies into the final jar package > > Can you elaborate on why that's not possible? > > Best regards, > > Martijn > > On Thu, Oct 27, 2022 at 6:59 AM Jacky Lau <liuyong...@gmail.com> wrote: > > > Hi guys: > > > > I'd like to initiate a discussion about adding command-line arguments to > > support user-dependent jar packages. > > > > Currently flink supports user's main jars through -jarfile or without > > setting this , the flink client will treat the first argument after that > as > > the user master jar package when it encounters the first command line > > argument that cannot be parsed. but it is not possible for platform users > > to create fat jars to package all their dependencies into the final jar > > package. In the meantime, the configuration pipeline.jars is currently > > exposed, and this value is overridden by command-line arguments such as > > -jarfile. > > > > And If the user is using both the command-line argument and the > > pipeline.jars argument, which can make the user werild and confused. In > > addition, we should specify the priority "command line parameter > -D > > dynamic parameter > flink-conf.yml configuration file parameter" in docs > > >
