[
https://issues.apache.org/jira/browse/FLINK-9762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16536737#comment-16536737
]
ASF GitHub Bot commented on FLINK-9762:
---------------------------------------
GitHub user JTaky opened a pull request:
https://github.com/apache/flink/pull/6284
[FLINK-9762] CoreOptions.TMP_DIRS ('io.tmp.dirs') wrongly managed on Yarn
## What is the purpose of the change
Currently TMP_DIRS (io.tmp.dirs) option has a bug in case of default
behaviour on Yarn platform:
If not value has been set via parameter on Yarn platform the env value
'LOCAL_DIRS' is used on Application Master.
When task manager is started this 'LOCAL_DIRS' is passed to the
TaskManager, so TaskManager do not understand that he needs to have a default
behaviour.
The proposed ~~hack~~ fix is to remove TMP_DIRS value in case if default
behaviour is required before starting TaskManager.
## Brief change log
* Extract method which set _io.tmp.dirs_ configuration value
* Fix configuration pass to TaskManager (**I am not sure what is the best
way to implement this**):
** Add a new flag, saying _io.tmp.dirs_ is value has been overridden
** If _io.tmp.dirs_ has *not* been overridden we remove _io.tmp.dirs_
remove config value, by setting the empty value (should we remove? clone
without a given key?)
* Correction of the documentation
## Verifying this change
This change is already covered by existing tests, such as
MiniClusterResource for Mesos part.
1. I do not see a way to test launching of the TaskManager with a correct
configuration.
2. A unit test documenting specific logic of TMP_DIRS managing is added:
BootstrapToolsTest#testSetTmpDirectoriesConfig
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): no
- The public API, i.e., is any changed class annotated with
`@Public(Evolving)`: no
- The serializers: no
- The runtime per-record code paths (performance sensitive): no
- Anything that affects deployment or recovery: JobManager (and its
components), Checkpointing, Yarn/Mesos, ZooKeeper: no
- The S3 file system connector: no
## Documentation
- Does this pull request introduce a new feature? no
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/JTaky/flink tmpDirs
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/6284.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #6284
----
commit d0c0bf7682843947bd1cfdf27775eb75c2864232
Author: Oleksandr Nitavskyi <o.nitavskyi@...>
Date: 2018-07-02T07:42:17Z
[FLINK-9762][yarn] Extract method: tmp dirs setting
commit 99bdeb924a2bf91eab39462ac8253ee79dcb1137
Author: Oleksandr Nitavskyi <o.nitavskyi@...>
Date: 2018-07-02T08:06:56Z
[FLINK-9762][yarn] Fix TMP_DIRS configuration passing
* HACK: Instead of removing set value as empty
commit 2fd5754209508e082e4678e8605a25e3d25f8209
Author: Oleksandr Nitavskyi <o.nitavskyi@...>
Date: 2018-07-09T09:15:14Z
[FLINK-9762] correct documenatation for 'io.tmp.dirs' config
----
> CoreOptions.TMP_DIRS wrongly managed on Yarn
> --------------------------------------------
>
> Key: FLINK-9762
> URL: https://issues.apache.org/jira/browse/FLINK-9762
> Project: Flink
> Issue Type: Bug
> Components: YARN
> Affects Versions: 1.5.0
> Reporter: Oleksandr Nitavskyi
> Assignee: vinoyang
> Priority: Major
> Labels: pull-request-available
>
> The issue on Yarn is that it is impossible to have different LOCAL_DIRS on
> JobManager and TaskManager, despite LOCAL_DIRS value depends on the container.
> The issue is that CoreOptions.TMP_DIRS is configured to the default value
> during JobManager initialization and added to the configuration object. When
> TaskManager is launched the appropriate configuration object is cloned with
> LOCAL_DIRS which makes sense only for Job Manager container. When YARN
> container with TaskManager from his point of view CoreOptions.TMP_DIRS is
> always equal either to path in flink.yml or to the or to the LOCAL_DIRS of
> Job Manager (default behaviour). Is TaskManager’s container do not have an
> access to another folders, that folders allocated by YARN TaskManager cannot
> be started.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
