[
https://issues.apache.org/jira/browse/FLINK-9366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16496516#comment-16496516
]
ASF GitHub Bot commented on FLINK-9366:
---------------------------------------
GitHub user dawidwys opened a pull request:
https://github.com/apache/flink/pull/6107
[FLINK-9366] DistributedCache works with Distributed File System
## What is the purpose of the change
Allows to use DistributeCache to cache files from cluster internal DFS
(reverted behaviour).
## Brief change log
*(for example:)*
- Local files (based on uri schema) are distributed via BlobServer
- Files in DFS are cached from DFS
## Verifying this change
* Added test for distributing files through DFS:
org.apache.flink.hdfstests.DistributedCacheDfsTest
* local files shipping covered by previous tests
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): no
- The public API, i.e., is any changed class annotated with
`@Public(Evolving)`:no
- The serializers: no
- The runtime per-record code paths (performance sensitive): no
- Anything that affects deployment or recovery: JobManager (and its
components), Checkpointing, Yarn/Mesos, ZooKeeper: no
- The S3 file system connector: no
## Documentation
- Does this pull request introduce a new feature? no
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/dawidwys/flink FLINK-9366
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/6107.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #6107
----
commit cc1344d18e2b1d9953e21d5189f2813fdabd7b01
Author: Dawid Wysakowicz <dwysakowicz@...>
Date: 2018-05-31T12:35:44Z
[FLINK-9366] DistributedCache works with Distributed File System
----
> Distribute Cache only works for client-accessible files
> -------------------------------------------------------
>
> Key: FLINK-9366
> URL: https://issues.apache.org/jira/browse/FLINK-9366
> Project: Flink
> Issue Type: Bug
> Components: Client, Local Runtime
> Affects Versions: 1.6.0
> Reporter: Chesnay Schepler
> Assignee: Dawid Wysakowicz
> Priority: Blocker
> Fix For: 1.6.0
>
>
> In FLINK-8620 the distributed cache was modified to the distribute files via
> the blob store, instead of downloading them from a distributed filesystem.
> Previously, taskmanagers would download requested files from the DFS. Now,
> they retrieve it form the blob store. This requires the client to
> preemptively upload all files used with distributed cache.
> As a result it is no longer possible to use the distributed cache for files
> that reside in a cluster-internal DFS, as the client cannot download it. This
> is a regression from the previous behavior and may break existing setups.
> [~aljoscha] [~dawidwys]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
