This command will help you download the user JARs without depending on the Filesystem CLI and init-container, as shown in the attached image. It handles the host path in Kubernetes, ensuring that only one process downloads the JARs while the other waits for it. This can also be used in a session cluster before starting JMs and TMs.
[image: image.png] Thanks Surendra On Tue, Mar 7, 2023 at 11:51 AM Surendra Singh Lilhore < surendralilh...@apache.org> wrote: > Hi David, > > Sorry I missed your reply. > > >>>Since you're mentioning docker, I assume you're deploying your > application to k8s. Is that correct? > > yes, I am using k8s. > > >>>For handcrafted Kubernetes deployments, you can simply download the > jar into the user lib folder in an init container [1]. You can usually > reuse existing docker images to download the jar. For example, for S3, > the AWS CLI image will do the trick [2]. > > I believe that since Flink already supports different filesystems, we can > utilize the same code without depending on any other CLI. This would help > us save the time currently taken by the init container. > > Not only can application mode clusters on Kubernetes utilize this, but > other clusters can also use it to download additional plugins before > starting the cluster. > > Thanks > Surendra > > > > > > On Sat, Mar 4, 2023 at 12:41 AM Surendra Singh Lilhore < > surendralilh...@apache.org> wrote: > >> Hi Team, >> >> >> >> According to the Flink documentation, in the APP mode, the application >> jar should be bundled with the Flink image. However, building an image for >> each new application can be difficult. Can we introduce new commands that >> will help to download the required jar locally before starting Flink JM or >> TM containers? This should be a simple command that depends on the >> supported file system (S3, HDFS, ABFS) in Flink, and the command format >> should be something like this: >> >> *./flink fs-download <Remote-FS Path> <Local-Path>* >> >> Example: >> >> *./flink fs-download >> abfs://mycontai...@storageaccount.dfs.core.windows.net/jars /tmp/appjars >> <http://mycontai...@storageaccount.dfs.core.windows.net/jars%0D/tmp/appjars>* >> >> I have already tested this in my cluster, and it is working fine. Before >> raising a JIRA ticket, I would like to get suggestions from the community. >> >> >> Thanks and Regards >> Surendra >> >
