Hi till, thanks for the feedback and suggestion.

I think it make senses to only support flink-dist-*.jar at the first step.
Just as your suggestion,
the config option could be "yarn.submission.automatic-flink-dist-upload",
default is true. Users
could use "-yt/--yarnship" to specify a HDFS path that contains
flink-dist-*.jar and set the above
config option to "false" to disable flink-dist-*.jar uploading.

Best,
Yang

Till Rohrmann <trohrm...@apache.org> 于2020年4月20日周一 下午8:07写道:

> Thanks for the clarification Yang. Now it makes sense to me.
>
> If it makes things easier, then I would still go first with the dead
> simple solution to turn automatic upload of local dist off via a
> configuration option before trying to implement a smart solution
> which relies on pattern matching or something else. For example, users
> might specify a remote location which is not accessible from the client.
> Then one could not figure out which files are already uploaded. The smart
> solution could be a follow up step then.
>
> Cheers,
> Till
>
> On Mon, Apr 20, 2020 at 1:09 PM Yang Wang <danrtsey...@gmail.com> wrote:
>
>> Hi till,
>>
>> Sorry for that i do not giving a detailed explanation of the
>> optimization. Actually, the optimization contains
>> the following two parts.
>> * Use remote uploaded jars to avoid unnecessary uploading(e.g.
>> flink-dist-*.jar, user jars, dependencies).
>> this could be done via enriching "-yt/--yarnship" to support remote ship
>> files.
>> * Use the "PUBLIC" or "PRIVATE" visibility of YARN local resource to
>> avoid unnecessary downloading. When
>> a local resource is public, once it is download by YARN NodeManager, it
>> could be reused by all the application
>> in the same NodeManager.
>>
>> >> Why do we need to specify the visibility of the remote files? Won't
>>> the visibility be specified when uploading these files?
>>
>> It is mostly for the users who want to eliminate the unnecessary
>> downloading so that the container could be
>> launched faster. "PRIVATE" means the remote jars could be shared by the
>> applications submitted by the current user.
>> "PUBLIC" means the remote jars could be shared by all the Flink
>> applications. And "APPLICATION" means they
>> could only be shared by the containers of the current application in a
>> same NodeManager.
>>
>>
>> For the implementation, i think we could do it step by step.
>> * Enrich "-yt/--yarnship" to support HDFS directory
>> * Add a new config option to control whether to avoid the unnecessary
>> uploading
>> * Enrich "-yt/--yarnship" to specify local resource visibility
>>
>>
>> Best,
>> Yang
>>
>>
>>
>> Till Rohrmann <trohrm...@apache.org> 于2020年4月20日周一 下午5:26写道:
>>
>>> Shall we say for the first version we only can deactivate the upload of
>>> local files instead of doing some optimizations? I guess my problem is that
>>> I don't fully understand the optimizations yet. Maybe we introduce a power
>>> user config option `yarn.submission.automatic-flink-dist-upload` or so.
>>>
>>> Why do we need to specify the visibility of the remote files? Won't the
>>> visibility be specified when uploading these files?
>>>
>>> Apart from that, the proposal looks good to me.
>>>
>>> Cheers,
>>> Till
>>>
>>> On Mon, Apr 20, 2020 at 5:38 AM Yang Wang <danrtsey...@gmail.com> wrote:
>>>
>>>> Hi tison,
>>>>
>>>> I think i get your concerns and points.
>>>>
>>>> Take both FLINK-13938[1] and FLINK-14964[2] into account, i will do in
>>>> the following steps.
>>>> * Enrich "-yt/--yarnship" to support HDFS directory
>>>> * Enrich "-yt/--yarnship" to specify local resource visibility. It is
>>>> "APPLICATION" by default. It could be also configured to "PUBLIC",
>>>> which means shared by all applications, or "PRIVATE" which means shared
>>>> by a same user.
>>>> * Add a new config option to control whether to optimize the
>>>> submission(default is false). When configured to true, Flink client will
>>>> try to filter the jars and files by name and size to avoid unnecessary
>>>> uploading.
>>>>
>>>> A very rough submission command could be issued as following.
>>>> *./bin/flink run -m yarn-cluster -d -yt
>>>> hdfs://myhdfs/flink/release/flink-1.11:PUBLIC,hdfs://myhdfs/user/someone/mylib
>>>> \*
>>>> *-yD yarn.submission-optimization.enable=true
>>>> examples/streaming/WindowJoin.jar*
>>>>
>>>> cc @Rong Rong <walter...@gmail.com>, since you also help to review the
>>>> old PR of FLINK-13938, maybe you could also share some thoughts.
>>>>
>>>>
>>>> [1]. https://issues.apache.org/jira/browse/FLINK-13938
>>>> [2]. https://issues.apache.org/jira/browse/FLINK-14964
>>>>
>>>>
>>>> Best,
>>>> Yang
>>>>
>>>>
>>>>
>>>> tison <wander4...@gmail.com> 于2020年4月18日周六 下午12:12写道:
>>>>
>>>>> Hi Yang,
>>>>>
>>>>> Name filtering & schema special handling makes sense for me. We can
>>>>> enrich later if there is requirement without breaking interface.
>>>>>
>>>>> For #1, from my perspective your first proposal is
>>>>>
>>>>>   having an option specifies remote flink/lib, then we turn off auto
>>>>> uploading local flink/lib and register that path as local resources
>>>>>
>>>>> It seems we here add another special logic for handling one kind of
>>>>> things...what I propose is we do these two steps explicitly separated:
>>>>>
>>>>> 1. an option turns off auto uploading local flink/lib
>>>>> 2. a general option register remote files as local resources
>>>>>
>>>>> The rest thing here is that you propose we handle flink/lib as PUBLIC
>>>>> visibility while other files as APPLICATION visibility, whether a
>>>>> composite configuration or name filtering to special handle libs makes
>>>>> sense though.
>>>>>
>>>>> YarnClusterDescriptor already has a lot of special handling logics
>>>>> which introduce a number of config options and keys, which should
>>>>> have been configured in few of common options and validated at the
>>>>> runtime.
>>>>>
>>>>> Best,
>>>>> tison.
>>>>>
>>>>>
>>>>> Yang Wang <danrtsey...@gmail.com> 于2020年4月17日周五 下午11:42写道:
>>>>>
>>>>>> Hi tison,
>>>>>>
>>>>>> For #3, if you mean registering remote HDFS file as local resource,
>>>>>> we should make the "-yt/--yarnship"
>>>>>> to support remote directory. I think it is the right direction.
>>>>>>
>>>>>> For #1, if the users could ship remote directory, then they could
>>>>>> also specify like this
>>>>>> "-yt hdfs://hdpdev/flink/release/flink-1.x,
>>>>>> hdfs://hdpdev/user/someone/mylib". Do you mean we add an
>>>>>> option for whether trying to avoid unnecessary uploading? Maybe we
>>>>>> could filter by names and file size.
>>>>>> I think this is a good suggestion, and we do not need to introduce a
>>>>>> new config option "-ypl".
>>>>>>
>>>>>> For #2, for flink-dist, the #1 could already solve the problem. We do
>>>>>> not need to support remote schema.
>>>>>> It will confuse the users when we only support HDFS, not S3, OSS, etc.
>>>>>>
>>>>>>
>>>>>> Best,
>>>>>> Yang
>>>>>>
>>>>>> tison <wander4...@gmail.com> 于2020年4月17日周五 下午8:05写道:
>>>>>>
>>>>>>> Hi Yang,
>>>>>>>
>>>>>>> I agree that these two of works would benefit from single assignee.
>>>>>>> My concern is as below
>>>>>>>
>>>>>>> 1. Both share libs & remote flink dist/libs are remote ship files. I
>>>>>>> don't think we have to implement multiple codepath/configuration.
>>>>>>> 2. So, for concept clarification, there are
>>>>>>>   (1) an option to disable shipping local libs
>>>>>>>   (2) flink-dist supports multiple schema at least we said "hdfs://"
>>>>>>>   (3) an option for registering remote shipfiles with path &
>>>>>>> visibility. I think new configuration system helps.
>>>>>>>
>>>>>>> the reason we have to special handling (2) instead of including it
>>>>>>> in (3) is because when shipping flink-dist to TM container, we specially
>>>>>>> detect flink-dist. Of course we can merge it into general ship files and
>>>>>>> validate shipfiles finally contain flink-dist, which is an alternative.
>>>>>>>
>>>>>>> The *most important* difference is (1) and (3) which we don't have
>>>>>>> an option for only remote libs. Is this clarification satisfy your 
>>>>>>> proposal?
>>>>>>>
>>>>>>> Best,
>>>>>>> tison.
>>>>>>>
>>>>>>>
>>>>>>> Till Rohrmann <trohrm...@apache.org> 于2020年4月17日周五 下午7:49写道:
>>>>>>>
>>>>>>>> Hi Yang,
>>>>>>>>
>>>>>>>> from what I understand it sounds reasonable to me. Could you sync
>>>>>>>> with Tison on FLINK-14964 on how to proceed. I'm not super deep into 
>>>>>>>> these
>>>>>>>> issues but they seem to be somewhat related and Tison already did some
>>>>>>>> implementation work.
>>>>>>>>
>>>>>>>> I'd say it be awesome if we could include this kind of improvement
>>>>>>>> into the release.
>>>>>>>>
>>>>>>>> Cheers,
>>>>>>>> Till
>>>>>>>>
>>>>>>>> On Thu, Apr 16, 2020 at 4:43 AM Yang Wang <danrtsey...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi All, thanks a lot for reviving this discussion.
>>>>>>>>>
>>>>>>>>> I think we could unify the FLINK-13938 and FLINK-14964 since they
>>>>>>>>> have the similar
>>>>>>>>> purpose, avoid unnecessary uploading and downloading jars in YARN
>>>>>>>>> deployment.
>>>>>>>>> The difference is FLINK-13938 aims to support the flink system lib
>>>>>>>>> directory only, while
>>>>>>>>> FLINK-14964 is trying to support arbitrary pre-uloaded
>>>>>>>>> jars(including user and system jars).
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> So i suggest to do this feature as following.
>>>>>>>>> 1. Upload the flink lib directory or users to hdfs, e.g.
>>>>>>>>> "hdfs://hdpdev/flink/release/flink-1.x"
>>>>>>>>> "hdfs://hdpdev/user/someone/mylib"
>>>>>>>>> 2. Use the -ypl argument to specify the shared lib, multiple
>>>>>>>>> directories could be specified
>>>>>>>>> 3. YarnClusterDescriptor will use the pre-uploaded jars to avoid
>>>>>>>>> unnecessary uploading,
>>>>>>>>> both for system and user jars
>>>>>>>>> 4. YarnClusterDescriptor needs to set the system jars to public
>>>>>>>>> visibility so that the distributed
>>>>>>>>> cache in the YARN nodemanager could be reused by multiple
>>>>>>>>> applications. This is to avoid
>>>>>>>>> unnecessary downloading, especially for the "flink-dist-*.jar".
>>>>>>>>> For the user shared lib, the
>>>>>>>>> visibility is still set to "APPLICATION" level.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> For our past internal use case, the shared lib could help with
>>>>>>>>> accelerating the submission a lot.
>>>>>>>>> Also it helps to reduce the pressure of HDFS when we want to
>>>>>>>>> launch many applications together.
>>>>>>>>>
>>>>>>>>> @tison @Till Rohrmann <trohrm...@apache.org> @Hailu, Andreas
>>>>>>>>> <andreas.ha...@gs.com> If you guys thinks the suggestion makes
>>>>>>>>> sense. I
>>>>>>>>> will try to find some time to work on this and hope it could catch
>>>>>>>>> up with release-1.1 cycle.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Best,
>>>>>>>>> Yang
>>>>>>>>>
>>>>>>>>> Hailu, Andreas [Engineering] <andreas.ha...@gs.com> 于2020年4月16日周四
>>>>>>>>> 上午8:47写道:
>>>>>>>>>
>>>>>>>>>> Okay, I’ll continue to watch the JIRAs. Thanks for the update,
>>>>>>>>>> Till.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> *// *ah
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> *From:* Till Rohrmann <trohrm...@apache.org>
>>>>>>>>>> *Sent:* Wednesday, April 15, 2020 10:51 AM
>>>>>>>>>> *To:* Hailu, Andreas [Engineering] <andreas.ha...@ny.email.gs.com
>>>>>>>>>> >
>>>>>>>>>> *Cc:* Yang Wang <danrtsey...@gmail.com>; tison <
>>>>>>>>>> wander4...@gmail.com>; user@flink.apache.org
>>>>>>>>>> *Subject:* Re: Flink Conf "yarn.flink-dist-jar" Question
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Hi Andreas,
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> it looks as if FLINK-13938 and FLINK-14964 won't make it into the
>>>>>>>>>> 1.10.1 release because the community is about to start the release 
>>>>>>>>>> process.
>>>>>>>>>> Since FLINK-13938 is a new feature it will be shipped with a major 
>>>>>>>>>> release.
>>>>>>>>>> There is still a bit of time until the 1.11 feature freeze and if 
>>>>>>>>>> Yang Wang
>>>>>>>>>> has time to finish this PR, then we could ship it.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Cheers,
>>>>>>>>>>
>>>>>>>>>> Till
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Wed, Apr 15, 2020 at 3:23 PM Hailu, Andreas [Engineering] <
>>>>>>>>>> andreas.ha...@gs.com> wrote:
>>>>>>>>>>
>>>>>>>>>> Yang, Tison,
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Do we know when some solution for 13938 and 14964 will arrive? Do
>>>>>>>>>> you think it will be in a 1.10.x version?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> *// *ah
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> *From:* Hailu, Andreas [Engineering]
>>>>>>>>>> *Sent:* Friday, March 20, 2020 9:19 AM
>>>>>>>>>> *To:* 'Yang Wang' <danrtsey...@gmail.com>
>>>>>>>>>> *Cc:* tison <wander4...@gmail.com>; user@flink.apache.org
>>>>>>>>>> *Subject:* RE: Flink Conf "yarn.flink-dist-jar" Question
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Hi Yang,
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> This is good to know. As a stopgap measure until a solution
>>>>>>>>>> between 13938 and 14964 arrives, we can automate the application 
>>>>>>>>>> staging
>>>>>>>>>> directory cleanup from our client should the process fail. It’s not 
>>>>>>>>>> ideal,
>>>>>>>>>> but will at least begin to manage our users’ quota. I’ll continue to 
>>>>>>>>>> watch
>>>>>>>>>> the two tickets. Thank you.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> *// *ah
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> *From:* Yang Wang <danrtsey...@gmail.com>
>>>>>>>>>> *Sent:* Monday, March 16, 2020 9:37 PM
>>>>>>>>>> *To:* Hailu, Andreas [Engineering] <andreas.ha...@ny.email.gs.com
>>>>>>>>>> >
>>>>>>>>>> *Cc:* tison <wander4...@gmail.com>; user@flink.apache.org
>>>>>>>>>> *Subject:* Re: Flink Conf "yarn.flink-dist-jar" Question
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Hi Hailu,
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Sorry for the late response. If the Flink cluster(e.g. Yarn
>>>>>>>>>> application) is stopped directly
>>>>>>>>>>
>>>>>>>>>> by `yarn application -kill`, then the staging directory will be
>>>>>>>>>> left behind. Since the jobmanager
>>>>>>>>>>
>>>>>>>>>> do not have any change to clean up the staging directly. Also it
>>>>>>>>>> may happen when the
>>>>>>>>>>
>>>>>>>>>> jobmanager crashed and reached the attempts limit of Yarn.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> For FLINK-13938, yes, it is trying to use the Yarn public cache
>>>>>>>>>> to accelerate the container
>>>>>>>>>>
>>>>>>>>>> launch.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Best,
>>>>>>>>>>
>>>>>>>>>> Yang
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Hailu, Andreas <andreas.ha...@gs.com> 于2020年3月10日周二 上午4:38写道:
>>>>>>>>>>
>>>>>>>>>> Also may I ask what causes these application ID directories to be
>>>>>>>>>> left behind? Is it a job failure, or can they persist even if the
>>>>>>>>>> application succeeds? I’d like to know so that I can implement my own
>>>>>>>>>> cleanup in the interim to prevent exceeding user disk space quotas.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> *// *ah
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> *From:* Hailu, Andreas [Engineering]
>>>>>>>>>> *Sent:* Monday, March 9, 2020 1:20 PM
>>>>>>>>>> *To:* 'Yang Wang' <danrtsey...@gmail.com>
>>>>>>>>>> *Cc:* tison <wander4...@gmail.com>; user@flink.apache.org
>>>>>>>>>> *Subject:* RE: Flink Conf "yarn.flink-dist-jar" Question
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Hi Yang,
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Yes, a combination of these two would be very helpful for us. We
>>>>>>>>>> have a single shaded binary which we use to run all of the jobs on 
>>>>>>>>>> our YARN
>>>>>>>>>> cluster. If we could designate a single location in HDFS for that as 
>>>>>>>>>> well,
>>>>>>>>>> we could also greatly benefit from FLINK-13938.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> It sounds like a general public cache solution is what’s being
>>>>>>>>>> called for?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> *// *ah
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> *From:* Yang Wang <danrtsey...@gmail.com>
>>>>>>>>>> *Sent:* Sunday, March 8, 2020 10:52 PM
>>>>>>>>>> *To:* Hailu, Andreas [Engineering] <andreas.ha...@ny.email.gs.com
>>>>>>>>>> >
>>>>>>>>>> *Cc:* tison <wander4...@gmail.com>; user@flink.apache.org
>>>>>>>>>> *Subject:* Re: Flink Conf "yarn.flink-dist-jar" Question
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Hi Hailu, tison,
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I created a very similar ticket before to accelerate Flink
>>>>>>>>>> submission on Yarn[1]. However,
>>>>>>>>>>
>>>>>>>>>> we do not get a consensus in the PR. Maybe it's time to revive
>>>>>>>>>> the discussion and try
>>>>>>>>>>
>>>>>>>>>> to find a common solution for both the two tickets[1][2].
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> [1]. https://issues.apache.org/jira/browse/FLINK-13938
>>>>>>>>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_FLINK-2D13938&d=DwMFaQ&c=7563p3e2zaQw0AB1wrFVgyagb2IE5rTZOYPxLxfZlX4&r=hRr4SA7BtUvKoMBP6VDhfisy2OJ1ZAzai-pcCC6TFXM&m=rlD0F8Cr4H0aPlN6O2_K13Q76RFOERSWuJANh4q6X_8&s=njA3vGYTf0g7Zsog8AiwS4bbXxblOxepBEWUV9W3E0s&e=>
>>>>>>>>>>
>>>>>>>>>> [2]. https://issues.apache.org/jira/browse/FLINK-14964
>>>>>>>>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_FLINK-2D14964&d=DwMFaQ&c=7563p3e2zaQw0AB1wrFVgyagb2IE5rTZOYPxLxfZlX4&r=hRr4SA7BtUvKoMBP6VDhfisy2OJ1ZAzai-pcCC6TFXM&m=rlD0F8Cr4H0aPlN6O2_K13Q76RFOERSWuJANh4q6X_8&s=9kT1RZkGwWh3MAbc_ZUrsEsmRRfw6VK4rlNIeNxs6GU&e=>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Best,
>>>>>>>>>>
>>>>>>>>>> Yang
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Hailu, Andreas <andreas.ha...@gs.com> 于2020年3月7日周六 上午11:21写道:
>>>>>>>>>>
>>>>>>>>>> Hi Tison, thanks for the reply. I’ve replied to the ticket. I’ll
>>>>>>>>>> be watching it as well.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> *// *ah
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> *From:* tison <wander4...@gmail.com>
>>>>>>>>>> *Sent:* Friday, March 6, 2020 1:40 PM
>>>>>>>>>> *To:* Hailu, Andreas [Engineering] <andreas.ha...@ny.email.gs.com
>>>>>>>>>> >
>>>>>>>>>> *Cc:* user@flink.apache.org
>>>>>>>>>> *Subject:* Re: Flink Conf "yarn.flink-dist-jar" Question
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> FLINK-13938 seems a bit different than your requirement. The one
>>>>>>>>>> totally matches is FLINK-14964
>>>>>>>>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_FLINK-2D14964&d=DwMFaQ&c=7563p3e2zaQw0AB1wrFVgyagb2IE5rTZOYPxLxfZlX4&r=hRr4SA7BtUvKoMBP6VDhfisy2OJ1ZAzai-pcCC6TFXM&m=9sMjDI0I_9Yni5ZWqV8GScK_KBTaA65yK9kBG-LE5_4&s=X1ZoN456fuc5mNxO6fBzDboEhrI0EHL873LzOd6tnN8&e=>.
>>>>>>>>>> I'll appreciate it if you can share you opinion on the JIRA ticket.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Best,
>>>>>>>>>>
>>>>>>>>>> tison.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> tison <wander4...@gmail.com> 于2020年3月7日周六 上午2:35写道:
>>>>>>>>>>
>>>>>>>>>> Yes your requirement is exactly taken into consideration by the
>>>>>>>>>> community. We currently have an open JIRA ticket for the specific
>>>>>>>>>> feature[1] and works for loosing the constraint of flink-jar schema 
>>>>>>>>>> to
>>>>>>>>>> support DFS location should happen.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Best,
>>>>>>>>>>
>>>>>>>>>> tison.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> [1] https://issues.apache.org/jira/browse/FLINK-13938
>>>>>>>>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_FLINK-2D13938&d=DwMFaQ&c=7563p3e2zaQw0AB1wrFVgyagb2IE5rTZOYPxLxfZlX4&r=hRr4SA7BtUvKoMBP6VDhfisy2OJ1ZAzai-pcCC6TFXM&m=9sMjDI0I_9Yni5ZWqV8GScK_KBTaA65yK9kBG-LE5_4&s=ediMPoQtcPX7K-5fjXJxE2cPp5OySkzwXYfYj8mDWO0&e=>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Hailu, Andreas <andreas.ha...@gs.com> 于2020年3月7日周六 上午2:03写道:
>>>>>>>>>>
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> We noticed that every time an application runs, it uploads the
>>>>>>>>>> flink-dist artifact to the /user/<user>/.flink HDFS directory. This 
>>>>>>>>>> causes
>>>>>>>>>> a user disk space quota issue as we submit thousands of apps to our 
>>>>>>>>>> cluster
>>>>>>>>>> an hour. We had a similar problem with our Spark applications where 
>>>>>>>>>> it
>>>>>>>>>> uploaded the Spark Assembly package for every app. Spark provides an
>>>>>>>>>> argument to use a location in HDFS its for applications to leverage 
>>>>>>>>>> so they
>>>>>>>>>> don’t need to upload them for every run, and that was our solution 
>>>>>>>>>> (see
>>>>>>>>>> “spark.yarn.jar” configuration if interested.)
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Looking at the Resource Orchestration Frameworks page
>>>>>>>>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__ci.apache.org_projects_flink_flink-2Ddocs-2Dstable_ops_config.html-23yarn-2Dflink-2Ddist-2Djar&d=DwMFaQ&c=7563p3e2zaQw0AB1wrFVgyagb2IE5rTZOYPxLxfZlX4&r=hRr4SA7BtUvKoMBP6VDhfisy2OJ1ZAzai-pcCC6TFXM&m=9sMjDI0I_9Yni5ZWqV8GScK_KBTaA65yK9kBG-LE5_4&s=3SPuvZu9nPph-qnE3TtbTngG-k3XDBLQGyk9I_tjNtI&e=>,
>>>>>>>>>> I see there’s might be a similar concept through a 
>>>>>>>>>> “yarn.flink-dist-jar”
>>>>>>>>>> configuration option. I wanted to place the flink-dist package we’re 
>>>>>>>>>> using
>>>>>>>>>> in a location in HDFS and configure out jobs to point to it, e.g.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> yarn.flink-dist-jar:
>>>>>>>>>> hdfs:////user/delp/.flink/flink-dist_2.11-1.9.1.jar
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Am I correct in that this is what I’m looking for? I gave this a
>>>>>>>>>> try with some jobs today, and based on what I’m seeing in the
>>>>>>>>>> launch_container.sh in our YARN application, it still looks like 
>>>>>>>>>> it’s being
>>>>>>>>>> uploaded:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> export
>>>>>>>>>> _FLINK_JAR_PATH="hdfs://d279536/user/delp/.flink/application_1583031705852_117863/flink-dist_2.11-1.9.1.jar"
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> How can I confirm? Or is this perhaps not config I’m looking for?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Best,
>>>>>>>>>>
>>>>>>>>>> Andreas
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> ------------------------------
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Your Personal Data: We may collect and process information about
>>>>>>>>>> you that may be subject to data protection laws. For more 
>>>>>>>>>> information about
>>>>>>>>>> how we use and disclose your personal data, how we protect your
>>>>>>>>>> information, our legal basis to use your information, your rights 
>>>>>>>>>> and who
>>>>>>>>>> you can contact, please refer to: www.gs.com/privacy-notices
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> ------------------------------
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Your Personal Data: We may collect and process information about
>>>>>>>>>> you that may be subject to data protection laws. For more 
>>>>>>>>>> information about
>>>>>>>>>> how we use and disclose your personal data, how we protect your
>>>>>>>>>> information, our legal basis to use your information, your rights 
>>>>>>>>>> and who
>>>>>>>>>> you can contact, please refer to: www.gs.com/privacy-notices
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> ------------------------------
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Your Personal Data: We may collect and process information about
>>>>>>>>>> you that may be subject to data protection laws. For more 
>>>>>>>>>> information about
>>>>>>>>>> how we use and disclose your personal data, how we protect your
>>>>>>>>>> information, our legal basis to use your information, your rights 
>>>>>>>>>> and who
>>>>>>>>>> you can contact, please refer to: www.gs.com/privacy-notices
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> ------------------------------
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Your Personal Data: We may collect and process information about
>>>>>>>>>> you that may be subject to data protection laws. For more 
>>>>>>>>>> information about
>>>>>>>>>> how we use and disclose your personal data, how we protect your
>>>>>>>>>> information, our legal basis to use your information, your rights 
>>>>>>>>>> and who
>>>>>>>>>> you can contact, please refer to: www.gs.com/privacy-notices
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> ------------------------------
>>>>>>>>>>
>>>>>>>>>> Your Personal Data: We may collect and process information about
>>>>>>>>>> you that may be subject to data protection laws. For more 
>>>>>>>>>> information about
>>>>>>>>>> how we use and disclose your personal data, how we protect your
>>>>>>>>>> information, our legal basis to use your information, your rights 
>>>>>>>>>> and who
>>>>>>>>>> you can contact, please refer to: www.gs.com/privacy-notices
>>>>>>>>>>
>>>>>>>>>

Reply via email to