Hi Oleksandr, I think your conclusions are correct. Thank you for looking into it. You can open a JIRA ticket describing the issue.
Best, Gary On Wed, Jul 4, 2018 at 9:30 AM, Oleksandr Nitavskyi <o.nitavs...@criteo.com> wrote: > Hello guys, > > We have discovered minor issue with Flink 1.5 on YARN particularly which > was related with the way Flink manages temp paths (io.tmp.dirs > > ) in configuration: https://ci.apache.org/projects/flink/flink-docs- > master/ops/config.html#io-tmp-dirs > > > > 1. From what we can see in the code, default option from > documentation doesn’t correspond to the reality on YARN or on Mesos > deployments. Looks like it equals to env variable ‘_FLINK_TMP_DIR’ on Mesos > and to `LOCAL_DIRS` on Yarn. > > 2. The issue on Yarn is that it is impossible to have different > LOCAL_DIRS on JobManager and TaskManager, despite LOCAL_DIRS value depends > on the container. > > The issue is that CoreOptions.TMP_DIRS is configured to the default value > during JobManager initialization and added to the configuration object. > When TaskManager is launched the appropriate configuration object is cloned > with LOCAL_DIRS which makes sense only for Job Manager container. When YARN > container with TaskManager from his point of view CoreOptions.TMP_DIRS is > always equal either to path in flink.yml or to the or to the LOCAL_DIRS of > Job Manager (default behaviour). Is TaskManager’s container do not have an > access to another folders, that folders allocated by YARN TaskManager > cannot be started. > > Could you please confirm that it is a bug and I will create a Jira ticket > to track it? > > Thanks > > Kind Regards > > Oleksandr Nitavskyi > > >
