Yes, it does interfere, I do have additional task managers. My batch application comes in my streaming cluster Flink’s GUI instead of creating its own container with its own GUI despite the –m yarn-cluster option.
De : Till Rohrmann [mailto:trohrm...@apache.org] Envoyé : jeudi 3 décembre 2015 10:36 À : user@flink.apache.org Objet : Re: HA Mode and standalone containers compatibility ? Hi Arnaud, as long as you don't have HA activated for your batch jobs, HA shouldn't have an influence on the batch execution. If it interferes, then you should see additional task manager connected to the streaming cluster when you execute the batch job. Could you check that? Furthermore, could you check that actually a second yarn application is started when you run the batch jobs? Cheers, Till On Thu, Dec 3, 2015 at 9:57 AM, LINZ, Arnaud <al...@bouyguestelecom.fr<mailto:al...@bouyguestelecom.fr>> wrote: Hello, I have both streaming applications & batch applications. Since the memory needs are not the same, I was using a long-living container for my streaming apps and new short-lived containers for hosting each batch execution. For that, I submit streaming jobs with "flink run" and batch jobs with "flink run -m yarn-cluster" This was working fine until I turned zookeeper HA mode on for my streaming applications. Even if I don't set it up in the yaml flink configuration file, but with -D options on the yarn_session.sh command line, now my batch jobs try to run in the streaming container, and fails because of the lack of ressources. My HA options are : -Dyarn.application-attempts=10 -Drecovery.mode=zookeeper -Drecovery.zookeeper.quorum=h1r1en01:2181 -Drecovery.zookeeper.path.root=/flink -Dstate.backend=filesystem -Dstate.backend.fs.checkpointdir=hdfs:///tmp/flink/checkpoints -Drecovery.zookeeper.storageDir=hdfs:///tmp/flink/recovery/ Am I missing something ? Best regards, Aranud ________________________________ L'intégrité de ce message n'étant pas assurée sur internet, la société expéditrice ne peut être tenue responsable de son contenu ni de ses pièces jointes. Toute utilisation ou diffusion non autorisée est interdite. Si vous n'êtes pas destinataire de ce message, merci de le détruire et d'avertir l'expéditeur. The integrity of this message cannot be guaranteed on the Internet. The company that sent this message cannot therefore be held liable for its content nor attachments. Any unauthorized use or dissemination is prohibited. If you are not the intended recipient of this message, then please delete it and notify the sender.
