Re: How to avoid path conflict in zookeeper/HDFS

2016-08-04 Thread Hironori Ogibayashi
Ufuk, Thank you for your answer. I understood only ZooKeeper root path should be different, and I am glad hear that YARN will automatically handle the root path in the next release. Regards, Hironori 2016-08-04 16:39 GMT+09:00 Ufuk Celebi : > Hey Hironori, > > the storage directories (recovery.z

Re: How to avoid path conflict in zookeeper/HDFS

2016-08-04 Thread Ufuk Celebi
Hey Hironori, the storage directories (recovery.zookeeper.storageDir, state.backend.fs.checkpointdir) can stay the same I think (either random or jobID-specific sub folders should be created there). The ZooKeeper root path (recovery.zookeeper.path.root) needs to be unique per cluster for HA. If y

How to avoid path conflict in zookeeper/HDFS

2016-08-03 Thread Hironori Ogibayashi
Hello, I have a question about Zookeeper or HDFS paths in case of running Flink on YARN. In my understanding, when I run multiple Flink cluster using the same zookeeper/HDFS, I have to specify different paths, as e.g. state.backend.fs.checkpointdir, recovery.zookeeper.path.root etc. If I run mul