[
https://issues.apache.org/jira/browse/FLINK-8279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16308077#comment-16308077
]
ASF GitHub Bot commented on FLINK-8279:
---------------------------------------
Github user NicoK commented on the issue:
https://github.com/apache/flink/pull/5176
@EronWright this PR originated from a user problem at
https://lists.apache.org/thread.html/177644ad38af34fb7ae7fb12a3bfcef3a70cd70dfccb6f7008a24eb6@%3Cuser.flink.apache.org%3E
And yes, this also adds the fallback to (a subdirectory in)
`taskmanager.tmp.dirs` in case no `blob.storage.directory` was given to the
JobManager. Given that the configuration parameter hints towards the
TaskManager only, we should maybe rename it?
Regarding
[MAPREDUCE-6472](https://issues.apache.org/jira/browse/MAPREDUCE-6472), it
seems this is only available from Hadoop 2.7.2 onwards. With this fix applied,
the behaviour changes only slightly: instead of using `java.io.tmpdir` directly
(set to `Environment.PWD/tmp`), this will use the `taskmanager.tmp.dirs` which
is set to `Environment.LOCAL_DIRS` by
`YarnTaskManagerRunner`/`YarnTaskExecutorRunner`.
I adapted the appropriate code paths for Mesos as well. If this is not used
yet, it will at least work in the future once the `taskmanager.tmp.dirs`
handling is correct. Until then, the fallback of this to `java.io.tmpdir` seems
to be working as well (if this is set appropriately by Mesos) - but this is not
only a problem of the BLOB server parts then and is out of the scope of this PR.
> Use Mesos/YARN temp directories as fallback for BlobServer/Cache temp
> directories
> ---------------------------------------------------------------------------------
>
> Key: FLINK-8279
> URL: https://issues.apache.org/jira/browse/FLINK-8279
> Project: Flink
> Issue Type: Improvement
> Components: Distributed Coordination, Network
> Affects Versions: 1.4.0
> Reporter: Nico Kruber
> Assignee: Nico Kruber
>
> Currently, the BLOB server and cache processes (temporarily) stash incoming
> files into their local file system in the directory given by the
> {{blob.storage.directory}} configuration property. If this property is not
> set or empty, it will fall back to {{java.io.tmpdir}}.
> Instead, in a Mesos/YARN environment, we could use the temporary directories
> they assigned to the Flink job which are not only the proper folder to use,
> but may also offer some more space.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
