[
https://issues.apache.org/jira/browse/FLINK-8279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16318112#comment-16318112
]
ASF GitHub Bot commented on FLINK-8279:
---------------------------------------
Github user NicoK commented on a diff in the pull request:
https://github.com/apache/flink/pull/5176#discussion_r160353839
--- Diff:
flink-runtime/src/main/java/org/apache/flink/runtime/blob/BlobUtils.java ---
@@ -127,21 +132,28 @@ private static BlobStoreService
createFileSystemBlobStore(Configuration configur
}
/**
- * Creates a local storage directory for a blob service under the given
parent directory.
+ * Creates a local storage directory for a blob service under the
configuration parameter given
+ * by {@link BlobServerOptions#STORAGE_DIRECTORY}. If this is
<tt>null</tt> or empty, we will
+ * fall back to Flink's temp directories (given by
+ * {@link org.apache.flink.configuration.CoreOptions#TMP_DIRS}) and
choose one among them at
+ * random.
*
- * @param basePath
- * base path, i.e. parent directory, of the storage
directory to use (if <tt>null</tt> or
- * empty, the path in <tt>java.io.tmpdir</tt> will be used)
+ * @param config
+ * Flink configuration
*
* @return a new local storage directory
*
* @throws IOException
* thrown if the local file storage cannot be created or
is not usable
*/
- static File initLocalStorageDirectory(String basePath) throws
IOException {
+ static File initLocalStorageDirectory(Configuration config) throws
IOException {
+
+ String basePath =
config.getString(BlobServerOptions.STORAGE_DIRECTORY);
+
File baseDir;
if (StringUtils.isNullOrWhitespaceOnly(basePath)) {
- baseDir = new
File(System.getProperty("java.io.tmpdir"));
+ final String[] tmpDirPaths =
TaskManagerServicesConfiguration.parseTempDirectories(config);
+ baseDir = new
File(tmpDirPaths[rnd.nextInt(tmpDirPaths.length)]);
--- End diff --
Unfortunately, the BLOB caches cannot handle multiple directories that
well. When looking for cached files, they check for their existence and
otherwise download missing ones. Looking through all of them may be cumbersome
as well (and should not be solved with this PR, imho).
I'd also expect most use cases to use a single directory but tbh, I lack
overview of use cases here.
> Use Mesos/YARN temp directories as fallback for BlobServer/Cache temp
> directories
> ---------------------------------------------------------------------------------
>
> Key: FLINK-8279
> URL: https://issues.apache.org/jira/browse/FLINK-8279
> Project: Flink
> Issue Type: Improvement
> Components: Distributed Coordination, Network
> Affects Versions: 1.4.0
> Reporter: Nico Kruber
> Assignee: Nico Kruber
>
> Currently, the BLOB server and cache processes (temporarily) stash incoming
> files into their local file system in the directory given by the
> {{blob.storage.directory}} configuration property. If this property is not
> set or empty, it will fall back to {{java.io.tmpdir}}.
> Instead, in a Mesos/YARN environment, we could use the temporary directories
> they assigned to the Flink job which are not only the proper folder to use,
> but may also offer some more space.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
