Sure, here is the cmdline output:
flink-1.3.2/bin/flink run -c com.corp.flink.KafkaJob quickstart.jar --topic
InputQ --bootstrap.servers localhost:9092 --zookeeper.connect localhost:2181
--group.id Consumers -p 5 --detached
Cluster configuration: Standalone cluster with JobManager at
localhost/127.0.0.1:6123
Using address localhost:6123 to connect to JobManager.
JobManager web interface address http://localhost:8081
Starting execution of program
Submitting job with JobID: 5cc74547361cec2ff9874764dac9ee91. Waiting for job
completion.
Connected to JobManager at
Actor[akka.tcp://flink@localhost:6123/user/jobmanager#-100905913] with leader
session id 00000000-0000-0000-0000-000000000000.
09/28/2017 16:30:40 Job execution switched to status RUNNING.
09/28/2017 16:30:40 Source: Custom Source -> Map(1/5) switched to SCHEDULED
09/28/2017 16:30:40 Source: Custom Source -> Map(2/5) switched to SCHEDULED
09/28/2017 16:30:40 Source: Custom Source -> Map(3/5) switched to SCHEDULED
09/28/2017 16:30:40 Source: Custom Source -> Map(4/5) switched to SCHEDULED
09/28/2017 16:30:40 Source: Custom Source -> Map(5/5) switched to SCHEDULED
09/28/2017 16:30:40 Sink: Unnamed(1/5) switched to SCHEDULED
09/28/2017 16:30:40 Sink: Unnamed(2/5) switched to SCHEDULED
09/28/2017 16:30:40 Sink: Unnamed(3/5) switched to SCHEDULED
09/28/2017 16:30:40 Sink: Unnamed(4/5) switched to SCHEDULED
09/28/2017 16:30:40 Sink: Unnamed(5/5) switched to SCHEDULED
...
I thought --detach will put the process in the background, and give me back the
cmdline, but maybe I got the meaning behind this option wrong?
Thank you!
>-------- Оригинално писмо --------
>От: Chesnay Schepler ches...@apache.org
>Относно: Re: how many 'run -c' commands to start?
>До: user@flink.apache.org
>Изпратено на: 29.09.2017 18:01
The only nodes that matter are those on which the Flink processes, i.e
Job- and TaskManagers, are being run.
To prevent a JobManager node failure from causing the job to fail you have
to look into an
HA setup.
(The jobmanager is responsible for distributing/coordinating work)
In case of a TaskManager node failure the job will fail and
restart, provided that enough TaskManagers are
still alive to satisfy the resource requirements of the job.
Can you elaborate a bit more what happened when you used the --detached
param?
On 28.09.2017 16:33, r. r. wrote:
Thank you, Chesnay
to make sure - should the node where the job has been submitted goes down, the
processing will continue, I hope?
Do I need to ensure this by configuration?
btw I added --detached param to the run cmd, but it didn't go into background
process as I would've expected. Am I guessing wrong?
Thanks!
Rob
>-------- Оригинално писмо --------
>От: Chesnay Schepler ches...@apache.org
>Относно: Re: how many 'run -c' commands to start?
>До: user@flink.apache.org
>Изпратено на: 28.09.2017 15:05
Hi!
Given a Flink cluster, you would only call `flink run ...` to submit a
job once; for simplicity i would submit it on the node where you started
the cluster. Flink will automatically distribute job across the cluster,
in smaller independent parts known as Tasks.
Regards,
Chesnay
On 28.09.2017 08:31, r. r. wrote:
Hello
I successfully ran a job with 'flink run -c', but this is for the local
setup.
How should i proceed with a cluster? Will flink automagically instantiate
the job on all servers - i hope i don't have to start 'flink run -c' on
all
machines.
New to flink and bigdata, so sorry for the probably silly question
Thanks!
Rob
