Hi Oleksandr, sorry for my late reply. This sounds indeed like a bug. Thanks for opening FLINK-9762 and the corresponding PR. I'll give it a review shortly so that we can include it in Flink 1.6.
Cheers, Till On Mon, Jul 2, 2018 at 2:31 PM Oleksandr Nitavskyi <o.nitavs...@criteo.com> wrote: > Hello guys, > > We have discovered minor issue with Flink 1.5 on YARN particularly which > was related with the way Flink manages temp paths (io.tmp.dirs > ) in configuration: > https://ci.apache.org/projects/flink/flink-docs-master/ops/config.html#io-tmp-dirs > > > 1. From what we can see in the code, default option doesn’t > correspond to reality on YARN or on Mesos deployments. Looks like it equals > to env variable ‘_FLINK_TMP_DIR’ on Mesos and to `LOCAL_DIRS` on Yarn. > > 2. The issue on Yarn is that it is impossible to have different > LOCAL_DIRS on JobManager and TaskManager, despite LOCAL_DIRS value depends > on the container. > > The issue is that CoreOptions.TMP_DIRS is configured to the default value > during JobManager initialization and added to the configuration object. > When TaskManager is launched the appropriate configuration object is cloned > with LOCAL_DIRS which makes sense only for Job Manager container. When YARN > container with TaskManager from his point of view CoreOptions.TMP_DIRS is > always equal either to path in flink.yml or to the or to the LOCAL_DIRS of > Job Manager (default behaviour). Is TaskManager’s container do not have an > access to another folders, that folders allocated by YARN TaskManager > cannot be started. > > Could you please confirm that it is a bug and I will create a Jira ticket > to track it? > > Thanks > Kind Regards > Oleksandr Nitavskyi > > >
