FLINK-13938 seems a bit different than your requirement. The one totally matches is FLINK-14964 <https://issues.apache.org/jira/browse/FLINK-14964>. I'll appreciate it if you can share you opinion on the JIRA ticket.
Best, tison. tison <wander4...@gmail.com> 于2020年3月7日周六 上午2:35写道： > Yes your requirement is exactly taken into consideration by the community. > We currently have an open JIRA ticket for the specific feature[1] and works > for loosing the constraint of flink-jar schema to support DFS location > should happen. > > Best, > tison. > > [1] https://issues.apache.org/jira/browse/FLINK-13938 > > > Hailu, Andreas <andreas.ha...@gs.com> 于2020年3月7日周六 上午2:03写道： > >> Hi, >> >> >> >> We noticed that every time an application runs, it uploads the flink-dist >> artifact to the /user/<user>/.flink HDFS directory. This causes a user disk >> space quota issue as we submit thousands of apps to our cluster an hour. We >> had a similar problem with our Spark applications where it uploaded the >> Spark Assembly package for every app. Spark provides an argument to use a >> location in HDFS its for applications to leverage so they don’t need to >> upload them for every run, and that was our solution (see “spark.yarn.jar” >> configuration if interested.) >> >> >> >> Looking at the Resource Orchestration Frameworks page >> <https://ci.apache.org/projects/flink/flink-docs-stable/ops/config.html#yarn-flink-dist-jar>, >> I see there’s might be a similar concept through a “yarn.flink-dist-jar” >> configuration option. I wanted to place the flink-dist package we’re using >> in a location in HDFS and configure out jobs to point to it, e.g. >> >> >> >> yarn.flink-dist-jar: hdfs:////user/delp/.flink/flink-dist_2.11-1.9.1.jar >> >> >> >> Am I correct in that this is what I’m looking for? I gave this a try with >> some jobs today, and based on what I’m seeing in the launch_container.sh in >> our YARN application, it still looks like it’s being uploaded: >> >> >> >> export >> _FLINK_JAR_PATH="hdfs://d279536/user/delp/.flink/application_1583031705852_117863/flink-dist_2.11-1.9.1.jar" >> >> >> >> How can I confirm? Or is this perhaps not config I’m looking for? >> >> >> >> Best, >> >> Andreas >> >> ------------------------------ >> >> Your Personal Data: We may collect and process information about you that >> may be subject to data protection laws. For more information about how we >> use and disclose your personal data, how we protect your information, our >> legal basis to use your information, your rights and who you can contact, >> please refer to: www.gs.com/privacy-notices >> >
