Hi, The batch job does not need to be HA. I stopped everything, cleaned the temp files, added -Drecovery.mode=standalone and it seems to work now ! Strange, but good for me for now.
Thanks, Arnaud -----Message d'origine----- De : Ufuk Celebi [mailto:u...@apache.org] Envoyé : jeudi 3 décembre 2015 11:11 À : user@flink.apache.org Objet : Re: HA Mode and standalone containers compatibility ? Hey Arnaud, thanks for reporting this. I think Till’s suggestion will help to debug this (checking whether a second YARN application has been started)… You don’t want to run the batch application in HA mode, correct? I sounds like the batch job is submitted with the same config keys. Could you start the batch job explicitly with -Drecovery.mode=standalone? If you do want the batch job to be HA as well, you have to configure separate Zookeeper root paths: recovery.zookeeper.path.root: /flink-streaming-1 # for the streaming session recovery.zookeeper.path.root: /flink-batch # for the batch session – Ufuk > On 03 Dec 2015, at 11:01, LINZ, Arnaud <al...@bouyguestelecom.fr> wrote: > > Yes, it does interfere, I do have additional task managers. My batch > application comes in my streaming cluster Flink’s GUI instead of creating its > own container with its own GUI despite the –m yarn-cluster option. > > De : Till Rohrmann [mailto:trohrm...@apache.org] Envoyé : jeudi 3 > décembre 2015 10:36 À : user@flink.apache.org Objet : Re: HA Mode and > standalone containers compatibility ? > > Hi Arnaud, > > as long as you don't have HA activated for your batch jobs, HA shouldn't have > an influence on the batch execution. If it interferes, then you should see > additional task manager connected to the streaming cluster when you execute > the batch job. Could you check that? Furthermore, could you check that > actually a second yarn application is started when you run the batch jobs? > > Cheers, > Till > > On Thu, Dec 3, 2015 at 9:57 AM, LINZ, Arnaud <al...@bouyguestelecom.fr> wrote: > Hello, > > > > I have both streaming applications & batch applications. Since the memory > needs are not the same, I was using a long-living container for my streaming > apps and new short-lived containers for hosting each batch execution. > > > > For that, I submit streaming jobs with "flink run" and batch jobs with > "flink run -m yarn-cluster" > > > > This was working fine until I turned zookeeper HA mode on for my streaming > applications. > > Even if I don't set it up in the yaml flink configuration file, but with -D > options on the yarn_session.sh command line, now my batch jobs try to run in > the streaming container, and fails because of the lack of ressources. > > > > My HA options are : > > -Dyarn.application-attempts=10 -Drecovery.mode=zookeeper > -Drecovery.zookeeper.quorum=h1r1en01:2181 > -Drecovery.zookeeper.path.root=/flink -Dstate.backend=filesystem > -Dstate.backend.fs.checkpointdir=hdfs:///tmp/flink/checkpoints > -Drecovery.zookeeper.storageDir=hdfs:///tmp/flink/recovery/ > > > > Am I missing something ? > > > > Best regards, > > Aranud > > > > L'intégrité de ce message n'étant pas assurée sur internet, la société > expéditrice ne peut être tenue responsable de son contenu ni de ses pièces > jointes. Toute utilisation ou diffusion non autorisée est interdite. Si vous > n'êtes pas destinataire de ce message, merci de le détruire et d'avertir > l'expéditeur. > > The integrity of this message cannot be guaranteed on the Internet. The > company that sent this message cannot therefore be held liable for its > content nor attachments. Any unauthorized use or dissemination is prohibited. > If you are not the intended recipient of this message, then please delete it > and notify the sender.
