Re: yarn ship from s3

Thank You Xintong, I will look for these updates in the near future. Regards, Vijay On Wed, May 26, 2021 at 6:40 PM Xintong Song wrote: > Hi Vijay, > > Currently, Flink only supports shipping files from the local machine where > job is submitted. > > There are tickets [1][2][3] tracking the eff

Re: yarn ship from s3

Hi Vijay, Currently, Flink only supports shipping files from the local machine where job is submitted. There are tickets [1][2][3] tracking the efforts that shipping files from remote paths, e.g., http, hdfs, etc. Once the efforts are done, adding s3 as an additional supported schema should be st

Re: yarn ship from s3

Hi Pohl, I tried to ship my property file. Example: *-yarn.ship-files s3://applib/xx/xx/1.0-SNAPSHOT/application.properties \* *Error:* 6:21:37.163 [main] ERROR org.apache.flink.client.cli.CliFrontend - Invalid command line arguments. org.apache.flink.client.cli.CliArgsException: Could not bui

Re: yarn ship from s3

Hi Vijay, have you tried yarn-ship-files [1] or yarn-ship-archives [2]? Maybe, that's what you're looking for... Best, Matthias [1] https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/deployment/config/#yarn-ship-files [2] https://ci.apache.org/projects/flink/flink-docs-release-1.13

Re: yarn ship from s3

Hi Piotr, I have been doing the same process as you mentioned so far, now I am migrating the deployment process using AWS CDK and AWS Step Functions, kind of like the CICD process. I added a download step of jar and configs (1, 2, 3 and 4) from S3 using command-runner.jar (AWS Step); it loaded tha

Re: yarn ship from s3

Hi Vijay, I'm not sure if I understand your question correctly. You have jar and configs (1, 2, 3 and 4) on S3 and you want to start a Flink job using those? Can you simply download those things (whole directory containing those) to the machine that will be starting the Flink job? Best, Piotrek

yarn ship from s3

Hi Team, I am trying to find a way to ship files from aws s3 for a flink streaming job, I am running on AWS EMR. What i need to ship are following: 1) application jar 2) application property file 3) custom flink-conf.yaml 4) log4j application specific Please let me know options. Thanks, Vijay

Re: FLINK YARN SHIP from S3 Directory

Thank You Gordon. I am still on 1.10.0. I will try 1.11.0 When I get there. Regards, Vijay On Fri, Sep 4, 2020 at 12:52 AM Tzu-Li (Gordon) Tai wrote: > Hi, > > As far as I can tell from a recent change [1], this seems to be possible > now starting from Flink 1.11.x. Have you already tried this

Re: FLINK YARN SHIP from S3 Directory

Hi, As far as I can tell from a recent change [1], this seems to be possible now starting from Flink 1.11.x. Have you already tried this with the latest Flink version? Also including Klou in this email, who might be able to confirm this. Cheers, Gordon [1] https://issues.apache.org/jira/browse/F

FLINK YARN SHIP from S3 Directory

Hi Team, Is there any feature to be able to ship directory to containers from s3 Directory instead of local. -yt,--yarnship Ship files in the specified directory (t for transfer)