Thanks, Chesnay, that was indeed the problem. It also explains why -p5 was not working for me from the cmdline
Best regards Robert >-------- Оригинално писмо -------- >От: Chesnay Schepler ches...@apache.org >Относно: Re: how many 'run -c' commands to start? >До: "r. r." <rob...@abv.bg> >Изпратено на: 01.10.2017 00:58 > The order in which you pass program arguments is quite important. Can > you try again putting --detached directly after run? > I.e. flink run --detached -c ... > > The reason being that arguments after the jar are treated as arguments > for the user-program, and not as arguments for the job execution. > > On 29.09.2017 17:23, r. r. wrote: > > Sure, here is the cmdline output: > > > > flink-1.3.2/bin/flink run -c com.corp.flink.KafkaJob quickstart.jar --topic > > InputQ --bootstrap.servers localhost:9092 --zookeeper.connect > > localhost:2181 --group.id Consumers -p 5 --detached > > Cluster configuration: Standalone cluster with JobManager at > > localhost/127.0.0.1:6123 > > Using address localhost:6123 to connect to JobManager. > > JobManager web interface address http://localhost:8081 > > Starting execution of program > > Submitting job with JobID: 5cc74547361cec2ff9874764dac9ee91. Waiting for > > job completion. > > Connected to JobManager at > > Actor[akka.tcp://flink@localhost:6123/user/jobmanager#-100905913] with > > leader session id 00000000-0000-0000-0000-000000000000. > > 09/28/2017 16:30:40 Job execution switched to status RUNNING. > > 09/28/2017 16:30:40 Source: Custom Source -> Map(1/5) switched to > > SCHEDULED > > 09/28/2017 16:30:40 Source: Custom Source -> Map(2/5) switched to > > SCHEDULED > > 09/28/2017 16:30:40 Source: Custom Source -> Map(3/5) switched to > > SCHEDULED > > 09/28/2017 16:30:40 Source: Custom Source -> Map(4/5) switched to > > SCHEDULED > > 09/28/2017 16:30:40 Source: Custom Source -> Map(5/5) switched to > > SCHEDULED > > 09/28/2017 16:30:40 Sink: Unnamed(1/5) switched to SCHEDULED > > 09/28/2017 16:30:40 Sink: Unnamed(2/5) switched to SCHEDULED > > 09/28/2017 16:30:40 Sink: Unnamed(3/5) switched to SCHEDULED > > 09/28/2017 16:30:40 Sink: Unnamed(4/5) switched to SCHEDULED > > 09/28/2017 16:30:40 Sink: Unnamed(5/5) switched to SCHEDULED > > ... > > > > I thought --detach will put the process in the background, and give me back > > the cmdline, but maybe I got the meaning behind this option wrong? > > > > Thank you! > > > > > > > > > > > > > > > > >-------- Оригинално писмо -------- > > > > >От: Chesnay Schepler ches...@apache.org > > > > >Относно: Re: how many 'run -c' commands to start? > > > > >До: user@flink.apache.org > > > > >Изпратено на: 29.09.2017 18:01 > > > > > > > > > > > >> > > > >> > > > >> > > > >> > > > >> The only nodes that matter are those on which the Flink processes, i.e > > > >> Job- and TaskManagers, are being run. > > > >> > > > >> To prevent a JobManager node failure from causing the job to fail you > >> have to look into an > > > >> HA setup. > > > >> (The jobmanager is responsible for distributing/coordinating work) > > > >> > > > >> In case of a TaskManager node failure the job will fail and > > > >> restart, provided that enough TaskManagers are > > > >> still alive to satisfy the resource requirements of the job. > > > >> > > > >> > > > >> Can you elaborate a bit more what happened when you used the > >> --detached param? > > > >> > > > >> On 28.09.2017 16:33, r. r. wrote: > > > >> > > > >> > > > >> > > > >> Thank you, Chesnay > > > >> to make sure - should the node where the job has been submitted goes down, > >> the processing will continue, I hope? > > > >> Do I need to ensure this by configuration? > > > > > >> btw I added --detached param to the run cmd, but it didn't go into > >> background process as I would've expected. Am I guessing wrong? > > > > > >> Thanks! > > > >> Rob > > > > > >> >-------- Оригинално писмо -------- > > > > > >> >От: Chesnay Schepler ches...@apache.org > > > > > >> >Относно: Re: how many 'run -c' commands to start? > > > > > >> >До: user@flink.apache.org > > > > > >> >Изпратено на: 28.09.2017 15:05 > > > > > >> > > > >> > > > >> > > > >> Hi! > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> Given a Flink cluster, you would only call `flink run ...` to submit a > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> job once; for simplicity i would submit it on the node where you > >> started > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> the cluster. Flink will automatically distribute job across the > >> cluster, > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> in smaller independent parts known as Tasks. > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> Regards, > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> Chesnay > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> On 28.09.2017 08:31, r. r. wrote: > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> Hello > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> I successfully ran a job with 'flink run -c', but this is for the > >> local > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> setup. > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> How should i proceed with a cluster? Will flink automagically > >> instantiate > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> the job on all servers - i hope i don't have to start 'flink run -c' > >> on all > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> machines. > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> New to flink and bigdata, so sorry for the probably silly question > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> Thanks! > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> Rob > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >>
