I’ve added a note about this to the docs and asked Max to trigger a new build of them.
Regarding Aljoscha’s idea: I like it. It is essentially a shortcut for configuring the root path. In any case, it is orthogonal to Till’s proposals. That one we need to address as well (see FLINK-2929). The motivation for the current behaviour was to be rather defensive when removing state in order to not loose data accidentally. But it can be confusing, indeed. – Ufuk > On 19 Nov 2015, at 12:08, Till Rohrmann <trohrm...@apache.org> wrote: > > You mean an additional start-up parameter for the `start-cluster.sh` script > for the HA case? That could work. > > On Thu, Nov 19, 2015 at 11:54 AM, Aljoscha Krettek <aljos...@apache.org> > wrote: > Maybe we could add a user parameter to specify a cluster name that is used to > make the paths unique. > > > On Thu, Nov 19, 2015, 11:24 Till Rohrmann <trohrm...@apache.org> wrote: > I agree that this would make the configuration easier. However, it entails > also that the user has to retrieve the randomized path from the logs if he > wants to restart jobs after the cluster has crashed or intentionally > restarted. Furthermore, the system won't be able to clean up old checkpoint > and job handles in case that the cluster stop was intentional. > > Thus, the question is how do we define the behaviour in order to retrieve > handles and to clean up old handles so that ZooKeeper won't be cluttered with > old handles? > > There are basically two modes: > > 1. Keep state handles when shutting down the cluster. Provide a mean to > define a fixed path when starting the cluster and also a mean to purge old > state handles. Furthermore, add a shutdown mode where the handles under the > current path are directly removed. This mode would guarantee to always have > the state handles available if not explicitly told differently. However, the > downside is that ZooKeeper will be cluttered most certainly. > > 2. Remove the state handles when shutting down the cluster. Provide a > shutdown mode where we keep the state handles. This will keep ZooKeeper clean > but will give you also the possibility to keep a checkpoint around if > necessary. However, the user is more likely to lose his state when shutting > down the cluster. > > On Thu, Nov 19, 2015 at 10:55 AM, Robert Metzger <rmetz...@apache.org> wrote: > I agree with Aljoscha. Many companies install Flink (and its config) in a > central directory and users share that installation. > > On Thu, Nov 19, 2015 at 10:45 AM, Aljoscha Krettek <aljos...@apache.org> > wrote: > I think we should find a way to randomize the paths where the HA stuff stores > data. If users don’t realize that they store data in the same paths this > could lead to problems. > > > On 19 Nov 2015, at 08:50, Till Rohrmann <trohrm...@apache.org> wrote: > > > > Hi Gwenhaël, > > > > good to hear that you could resolve the problem. > > > > When you run multiple HA flink jobs in the same cluster, then you don’t > > have to adjust the configuration of Flink. It should work out of the box. > > > > However, if you run multiple HA Flink cluster, then you have to set for > > each cluster a distinct ZooKeeper root path via the option > > recovery.zookeeper.path.root in the Flink configuraiton. This is necessary > > because otherwise all JobManagers (the ones of the different clusters) will > > compete for a single leadership. Furthermore, all TaskManagers will only > > see the one and only leader and connect to it. The reason is that the > > TaskManagers will look up their leader at a ZNode below the ZooKeeper root > > path. > > > > If you have other questions then don’t hesitate asking me. > > > > Cheers, > > Till > > > > > > On Wed, Nov 18, 2015 at 6:37 PM, Gwenhael Pasquiers > > <gwenhael.pasqui...@ericsson.com> wrote: > > Nevermind, > > > > > > > > Looking at the logs I saw that it was having issues trying to connect to ZK. > > > > To make I short is had the wrong port. > > > > > > > > It is now starting. > > > > > > > > Tomorrow I’ll try to kill some JobManagers *evil*. > > > > > > > > Another question : if I have multiple HA flink jobs, are there some points > > to check in order to be sure that they won’t collide on hdfs or ZK ? > > > > > > > > B.R. > > > > > > > > Gwenhaël PASQUIERS > > > > > > > > From: Till Rohrmann [mailto:till.rohrm...@gmail.com] > > Sent: mercredi 18 novembre 2015 18:01 > > To: user@flink.apache.org > > Subject: Re: YARN High Availability > > > > > > > > Hi Gwenhaël, > > > > > > > > do you have access to the yarn logs? > > > > > > > > Cheers, > > > > Till > > > > > > > > On Wed, Nov 18, 2015 at 5:55 PM, Gwenhael Pasquiers > > <gwenhael.pasqui...@ericsson.com> wrote: > > > > Hello, > > > > > > > > We’re trying to set up high availability using an existing zookeeper quorum > > already running in our Cloudera cluster. > > > > > > > > So, as per the doc we’ve changed the max attempt in yarn’s config as well > > as the flink.yaml. > > > > > > > > recovery.mode: zookeeper > > > > recovery.zookeeper.quorum: host1:3181,host2:3181,host3:3181 > > > > state.backend: filesystem > > > > state.backend.fs.checkpointdir: hdfs:///flink/checkpoints > > > > recovery.zookeeper.storageDir: hdfs:///flink/recovery/ > > > > yarn.application-attempts: 1000 > > > > > > > > Everything is ok as long as recovery.mode is commented. > > > > As soon as I uncomment recovery.mode the deployment on yarn is stuck on : > > > > > > > > “Deploying cluster, current state ACCEPTED”. > > > > “Deployment took more than 60 seconds….” > > > > Every second. > > > > > > > > And I have more than enough resources available on my yarn cluster. > > > > > > > > Do you have any idea of what could cause this, and/or what logs I should > > look for in order to understand ? > > > > > > > > B.R. > > > > > > > > Gwenhaël PASQUIERS > > > > > > > > > > > >
