Hi Julio, I agree that the job submission should work in HA mode if you manually specify the JobManager. At the minimum a proper error message should be shown. Feel free to open an issue in JIRA.
You already stated that you can maintain multiple configuration directories as a workaround. It is possible to switch between them by setting the FLINK_CONF_DIR environment variable, e.g, FLINK_CONF_DIR=/path/to/conf-dir-1 bin/flink run ... FLINK_CONF_DIR=/path/to/conf-dir-2 bin/flink run ... Beginning from 1.5 this should be a non-issue because the job submission happens through HTTP and every non-leading master redirects requests to the leading master. Best, Gary On Thu, May 3, 2018 at 10:23 PM, Julio Biason <julio.bia...@azion.com> wrote: > Hey Gary (again), > > Yup, that worked. Now I can launch apps again. > > ... but that's not something actually good. > > I mean, I have my own test environment, which doesn't need HA -- after > all, I don't need to worry about this, this is a framework job, not my > pipeline job. Which means now I'll need to either keep two different > configuration files and keep switching between them -- because the `flink` > command doesn't not accept a configuration file (or, at least, it's not > listed on `--help`) or I'll have to first copy to the prod/staging machines > and then run there -- which seems a waste, since it seems using `flink run > -m` already adds the file in the blob server and then runs and, doing the > copy-to-machine step means there are two copies going on. > > I mean, if I say "hey flink, run this job _there_", flink should be smart > enough to read how "there" is running things and adjust itself. The > environment which started the run may not follow the same rules as the > target run machine. > > ... and, in the end, it seems this is mostly useless discussion, as > JobManager is changing completely on 1.5 -- but I kinda worry if I will > have the same issue with the new ResourceManager... > > On Thu, May 3, 2018 at 11:00 AM, Julio Biason <julio.bia...@azion.com> > wrote: > >> Hey Gary, >> >> Yes, I was still running with the `-m` flag on my dev machine -- >> partially configured like prod, but without the HA stuff. I never thought >> it could be a problem, since even the web interface can redirect from the >> secondary back to primary. >> >> Currently I'm still running 1.4.0 (and I plan to upgrade to 1.4.2 as soon >> as I can fix this). >> >> I'll try again with the HA/ZooKeeper properly set up on my machine and, >> if it still balks, I'll send the (updated) logs. >> >> On Thu, May 3, 2018 at 9:36 AM, Gary Yao <g...@data-artisans.com> wrote: >> >>> Hi Julio, >>> >>> Are you using the -m flag of "bin/flink run" by any chance? In HA mode, >>> you >>> cannot manually specify the JobManager address. The client determines >>> the leader >>> through ZooKeeper. If you did not configure the ZooKeeper quorum in the >>> flink-conf.yaml on the machine from which you are submitting, this might >>> explain >>> the error message. >>> >>> > But that didn't solve my problem. So far, the `flink run` still fails >>> with the same message (I'm adding the full stacktrace of the failure in the >>> end, just in case), but now I'm also seeing this message in the JobManager >>> logs: >>> Unfortunately, the error message in your previous email is different. If >>> the >>> above does not solve your problem, can you attach the logs of the client >>> and >>> JobManager? >>> >>> Lastly, what Flink version are you running? >>> >>> Best, >>> Gary >>> >>> On Wed, May 2, 2018 at 6:51 PM, Julio Biason <julio.bia...@azion.com> >>> wrote: >>> >>>> Hey guys and gals, >>>> >>>> So, after a bit more digging, I found out that once HA is enabled, >>>> `jobmanager.rpc.port` is also ignore (along with `jobmanager.rpc.address`, >>>> but I was expecting this). Because I set the >>>> `high-availability.jobmanager.port` >>>> to `50010-50015`, my RPC port also changed (the docs made me think this >>>> would only affect the HA communication, not ALL communications). This can >>>> be checked on the Dashboard, under the JobManager configuration option. >>>> >>>> But that didn't solve my problem. So far, the `flink run` still fails >>>> with the same message (I'm adding the full stacktrace of the failure in the >>>> end, just in case), but now I'm also seeing this message in the JobManager >>>> logs: >>>> >>>> 2018-05-02 16:44:32,373 WARN org.apache.flink.runtime.jobma >>>> nager.JobManager - Discard message >>>> LeaderSessionMessage(00000000-0000-0000-0000-000000000000,SubmitJob(JobGraph(jobId: >>>> 42a25752ab085117a21c02d3db54777e),DETACHED)) because the expected >>>> leader session ID c01eba4f-44e2-4c65-85d5-a9a05ceba28e did not equal >>>> the received leader session ID 00000000-0000-0000-0000-000000 >>>> 000000. >>>> >>>> >>>> So, I'm still lost on where to go forward. >>>> >>>> >>>> Failure when using `flink run`: >>>> >>>> org.apache.flink.client.program.ProgramInvocationException: The >>>> program execution failed: JobManager did not respond within 60000 >>>> ms >>>> >>>> at org.apache.flink.client.program.ClusterClient.runDetached(Cl >>>> usterClient.java:524) >>>> at org.apache.flink.client.program.StandaloneClusterClient.subm >>>> itJob(StandaloneClusterClient.java:103) >>>> at org.apache.flink.client.program.ClusterClient.run(ClusterCli >>>> ent.java:456) >>>> at org.apache.flink.client.program.DetachedEnvironment.finalize >>>> Execute(DetachedEnvironment.java:77) >>>> at org.apache.flink.client.program.ClusterClient.run(ClusterCli >>>> ent.java:402) >>>> at org.apache.flink.client.CliFrontend.executeProgram(CliFronte >>>> nd.java:802) >>>> at org.apache.flink.client.CliFrontend.run(CliFrontend.java:282 >>>> ) >>>> at org.apache.flink.client.CliFrontend.parseParameters(CliFront >>>> end.java:1054) >>>> at org.apache.flink.client.CliFrontend$1.call(CliFrontend.java: >>>> 1101) >>>> at org.apache.flink.client.CliFrontend$1.call(CliFrontend.java: >>>> 1098) >>>> at java.security.AccessController.doPrivileged(Native Method) >>>> at javax.security.auth.Subject.doAs(Subject.java:422) >>>> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGro >>>> upInformation.java:1698) >>>> at org.apache.flink.runtime.security.HadoopSecurityContext.runS >>>> ecured(HadoopSecurityContext.java:41) >>>> at org.apache.flink.client.CliFrontend.main(CliFrontend.java:10 >>>> 98) >>>> Caused by: org.apache.flink.runtime.client.JobTimeoutException: >>>> JobManager did not respond within 60000 ms >>>> at org.apache.flink.runtime.client.JobClient.submitJobDetached( >>>> JobClient.java:437) >>>> at org.apache.flink.client.program.ClusterClient.runDetached(Cl >>>> usterClient.java:516) >>>> ... 14 more >>>> Caused by: java.util.concurrent.TimeoutException >>>> at java.util.concurrent.CompletableFuture.timedGet(CompletableF >>>> uture.java:1771) >>>> at java.util.concurrent.CompletableFuture.get(CompletableFuture >>>> .java:1915) >>>> at org.apache.flink.runtime.client.JobClient.submitJobDetached( >>>> JobClient.java:435) >>>> ... 15 more >>>> >>>> >>>> On Wed, May 2, 2018 at 9:52 AM, Julio Biason <julio.bia...@azion.com> >>>> wrote: >>>> >>>>> Hello all, >>>>> >>>>> I'm building a standalone cluster with HA JobManager. So far, >>>>> everything seems to work, but when i try to `flink run` my job, it fails >>>>> with the following error: >>>>> >>>>> Caused by: org.apache.flink.runtime.leade >>>>> rretrieval.LeaderRetrievalException: Could not retrieve the leader >>>>> gateway. >>>>> >>>>> So far, I have two different machines running the JobManager and, >>>>> looking at the logs, I can't see any problem whatsoever to explain why the >>>>> flink command is refusing to run the pipeline... >>>>> >>>>> Any ideas where I should look? >>>>> >>>>> -- >>>>> *Julio Biason*, Sofware Engineer >>>>> *AZION* | Deliver. Accelerate. Protect. >>>>> Office: +55 51 3083 8101 <callto:+555130838101> | Mobile: +55 51 >>>>> <callto:+5551996209291>*99907 0554* >>>>> >>>> >>>> >>>> >>>> -- >>>> *Julio Biason*, Sofware Engineer >>>> *AZION* | Deliver. Accelerate. Protect. >>>> Office: +55 51 3083 8101 <callto:+555130838101> | Mobile: +55 51 >>>> <callto:+5551996209291>*99907 0554* >>>> >>> >>> >> >> >> -- >> *Julio Biason*, Sofware Engineer >> *AZION* | Deliver. Accelerate. Protect. >> Office: +55 51 3083 8101 <callto:+555130838101> | Mobile: +55 51 >> <callto:+5551996209291>*99907 0554* >> > > > > -- > *Julio Biason*, Sofware Engineer > *AZION* | Deliver. Accelerate. Protect. > Office: +55 51 3083 8101 <callto:+555130838101> | Mobile: +55 51 > <callto:+5551996209291>*99907 0554* >
