I see... Because of the start-cluster script, I was under the impression that the jobmanager had to connect to each node upon start-up, which would make scaling an issue without restarting the job manager, but it makes sense now. Thanks for the clarification.
Side question:what happens if the job manager fails? Taskmanagers keep running the jobs? Should a typical production setup include multiple jobmanagers for replication, and if so how is that configured? Emmanuel -------- Original message -------- From: Stephan Ewen <[email protected]> Date:03/16/2015 1:09 AM (GMT-08:00) To: [email protected] Subject: Re: Scaling a Flink cluster Hi Emmanuel! The slaves file is not needed on every node. It is only used by the "start-cluster.sh" Script, which makes an ssh call to every host in that file to start a taskmanager. You can add a taskmanager to an existing flink cluster by simply calling "taskmanager.sh start" on that machine (which should have a flink-conf.yaml file). The flink-conf.yaml may actually be different for every taskmanager as well, but that is a more uncommon setup... Greetings, Stephan Am 16.03.2015 08:27 schrieb "Emmanuel" <[email protected]>: > Hello, > > In my understanding, the flink-conf.yaml is the one config file to > configure a cluster. > The slave file lists the slave nodes. > they must both be on every node. > > I'm trying to understand what is the best strategy to scale a Flink > cluster since: > - adding a node means adding an entry to the slave list and replicating on > every node. > > Does the cluster need to be restarted to take the new nodes into account? > It seems like it. > Having to replicate the file on all nodes is not super convenient. > Restarting is even more trouble. > Is there a way to scale a live cluster? If so how? > Any link to the info would be helpful. > > Thanks >
