Hey Arnaud,
The cause of this is probably that both jobs use the same ZooKeeper
root path, in which case all task managers connect to the same leading
job manager.
I think you forgot to the add the y in the -Drecovery.mode=standalone
for the batch jobs, e.g.
-yDrecovery.mode=standalone
Can you try this?
– Ufuk
On Mon, Jun 6, 2016 at 2:19 PM, LINZ, Arnaud <al...@bouyguestelecom.fr> wrote:
> Hi,
>
>
>
> I use Flink 1.0.0. I have a persistent yarn container set (a persistent
> flink job manager) that I use for streaming jobs ; and I use the
> “yarn-cluster” mode to launch my batches.
>
>
>
> I’ve just switched “HA” mode on for my streaming persistent job manager and
> it seems to works ; however my batches are not working any longer because
> they now execute themselves inside the persistent container (and fail
> because it lacks slots) and not in a separate standalone job manager.
>
>
>
> My batch launch options:
>
>
>
> CONTAINER_OPTIONS="-m yarn-cluster -yn $FLINK_NBCONTAINERS -ytm
> $FLINK_MEMORY -yqu $FLINK_QUEUE -ys $FLINK_NBSLOTS -yD
> yarn.heap-cutoff-ratio=$FLINK_HEAP_CUTOFF_RATIO -yD akka.ask.timeout=300s"
>
> JVM_ARGS="${JVM_ARGS} -Drecovery.mode=standalone
> -Dyarn.properties-file.location=/tmp/flink/batch"
>
>
>
> $FLINK_DIR/flink run $CONTAINER_OPTIONS --class $MAIN_CLASS_KUBERA $JAR_SUPP
> $listArgs $ACTION
>
>
>
> My persistent cluster launch option :
>
>
>
> export FLINK_HA_OPTIONS="-Dyarn.application-attempts=10
> -Drecovery.mode=zookeeper
> -Drecovery.zookeeper.quorum=${FLINK_HA_ZOOKEEPER_SERVERS}
> -Drecovery.zookeeper.path.root=${FLINK_HA_ZOOKEEPER_PATH}
> -Dstate.backend=filesystem
> -Dstate.backend.fs.checkpointdir=hdfs:///tmp/${FLINK_HA_ZOOKEEPER_PATH}/checkpoints
> -Drecovery.zookeeper.storageDir=hdfs:///tmp/${FLINK_HA_ZOOKEEPER_PATH}/recovery/"
>
>
>
> $FLINK_DIR/yarn-session.sh -Dyarn.heap-cutoff-ratio=$FLINK_HEAP_CUTOFF_RATIO
> $FLINK_HA_OPTIONS -st -d -n $FLINK_NBCONTAINERS -s $FLINK_NBSLOTS -tm
> $FLINK_MEMORY -qu $FLINK_QUEUE -nm ${GANESH_TYPE_PF}_KuberaFlink
>
>
>
> I’ve switched back to the FLINK_HA_OPTIONS="" way of launching the container
> for now, but I lack HA.
>
>
>
> Is it a (un)known bug or am I missing a magic option?
>
>
>
> Best regards,
>
> Arnaud
>
>
>
>
> ________________________________
>
> L'intégrité de ce message n'étant pas assurée sur internet, la société
> expéditrice ne peut être tenue responsable de son contenu ni de ses pièces
> jointes. Toute utilisation ou diffusion non autorisée est interdite. Si vous
> n'êtes pas destinataire de ce message, merci de le détruire et d'avertir
> l'expéditeur.
>
> The integrity of this message cannot be guaranteed on the Internet. The
> company that sent this message cannot therefore be held liable for its
> content nor attachments. Any unauthorized use or dissemination is
> prohibited. If you are not the intended recipient of this message, then
> please delete it and notify the sender.
