strange behavior with jobmanager.rpc.address on standalone HA cluster

[Derek VerLee]

Two things: 1. It would be beneficial I think to drop a line somewhere in the docs (probably on the production ready checklist as well as the HA page) explaining that enabling zookeeper "highavailability" allows for your jobs to restart automatically after a jobmanager crash or restart.  We had spent some cycles trying to implement job restarting and watchdogs (poorly) when I discoverd this from a flink forward presentation on youtube. 2. I seem to have found some odd behavior with HA and then found something that works, but I can't explain why.  The clifnotes version is that I took an existing standalone cluster with a single JM and modified with high availability zookeeper mode.  The same flink-conf.yaml file is used on all nodes (including JM). This seemed to work fine, I restarted the JM (jm0) and the jobs relaunched when it came back.  Easy!  Then I deployed a second JM (jm1).  Once I modified `masters`, set the HA rpc port range and opened those ports on the firewall for both jobmanagers, but left `jobmanager.rpc.address` the original value, `jm0` on all nodes.  I then observed that jm0 worked fine, taskmanagers connected to it and jobs ran.  jm1 did not 301 me to jm0 however, it displayed a dashboard (no jobs, no tm).  When I stopped jm0, the jobs show up on jm1 as RESTARTING, but the taskmanagers never attach to jm1.   In the logs, all nodes, including jm1, had messages about trying to reach jm0.  From the documentation and various comments I've seen, `jobmanager.rpc.address` should be ignored.  However, commenting it out entirely lead to jobmanagers crashing at boot, setting to `localhost` caused all the taskmanagers to log messages about trying to connect to the jobmanager at localhost.  What finally worked was to set the value to the hostname where the flink-conf.yaml was individually, even on the taskmanagers.  Does this seem like a bug? Just a hunch, but is there something called an "akka leader" that is different from the jobmanager leader, and could it be somehow defaulting its value over to jobmanager.rpc.address?