Shouldn't the non-HA case be covered by rest.address?
On 20.06.2018 09:40, Till Rohrmann wrote:
Hi Sampath,
it is no longer possible to not start the rest server endpoint by
setting rest.port to -1. If you do this, then the cluster won't start.
The comment in the flink-conf.yaml holds only true for the legacy mode.
In non-HA setups we need the jobmanager.rpc.address to derive the
hostname of the rest server. The jobmanager.rpc.port is no longer
needed for the client but only for the other cluster components (TMs).
When using the HA mode, then every address will be retrieved from
ZooKeeper.
I hope this clarifies things.
Cheers,
Till
On Wed, Jun 20, 2018 at 9:24 AM Chesnay Schepler <ches...@apache.org
<mailto:ches...@apache.org>> wrote:
I was worried this might be the case.
The rest.port handling was simply copied from the legacy web-server,
which explicitly allowed shutting it down.
It may (I'm not entirely sure) also not be necessary for all
deployment
modes; for example if the job is baked into the job/taskmanager
images.
I'm not quite sure whether the rpc address is actually required
for the
REST job submission, or only since we still rely partly on some
legacy
code (ClusterClient). Maybe Till (cc) knows the answer to that.
> Adding on to this point you made - " the rpc address is still
*required *due
> to some technical implementations; it may be that you can set
this to some
> arbitrary value however."
>
> For job submission to happen successfully we should give
specific rpc
> address and not any arbitrary value. If any arbitrary value is
given the
> job submission fails with the following error -
> org.apache.flink.client.deployment.ClusterRetrieveException:
Couldn't
> retrieve standalone cluster
> at
>
org.apache.flink.client.deployment.StandaloneClusterDescriptor.retrieve(StandaloneClusterDescriptor.java:51)
> at
>
org.apache.flink.client.deployment.StandaloneClusterDescriptor.retrieve(StandaloneClusterDescriptor.java:31)
> at
>
org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:249)
> at
org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:210)
> at
>
org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1020)
> at
>
org.apache.flink.client.cli.CliFrontend.lambda$main$9(CliFrontend.java:1096)
> at
>
org.apache.flink.runtime.security.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:30)
> at
> org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1096)
> Caused by: java.net.UnknownHostException:
flinktest-flink-jobmanager1233445:
> Name or service not known
> (Random name flinktest-flink-jobmanager1233445)
> at java.net.Inet6AddressImpl.lookupAllHostAddr(Native
Method)
> at
java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928)
> at
>
java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1323)
> at
java.net.InetAddress.getAllByName0(InetAddress.java:1276)
> at java.net.InetAddress.getAllByName(InetAddress.java:1192)
> at java.net.InetAddress.getAllByName(InetAddress.java:1126)
> at java.net.InetAddress.getByName(InetAddress.java:1076)
> at
>
org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils.getRpcUrl(AkkaRpcServiceUtils.java:171)
> at
>
org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils.getRpcUrl(AkkaRpcServiceUtils.java:136)
> at
>
org.apache.flink.runtime.highavailability.HighAvailabilityServicesUtils.createHighAvailabilityServices(HighAvailabilityServicesUtils.java:83)
> at
>
org.apache.flink.client.program.ClusterClient.<init>(ClusterClient.java:158)
> at
>
org.apache.flink.client.program.rest.RestClusterClient.<init>(RestClusterClient.java:184)
> at
>
org.apache.flink.client.program.rest.RestClusterClient.<init>(RestClusterClient.java:157)
> at
>
org.apache.flink.client.deployment.StandaloneClusterDescriptor.retrieve(StandaloneClusterDescriptor.java:49)
> ... 7 more
>
>
> On Wed, Jun 20, 2018 at 11:18 AM, Sampath Bhat
<sam414255p...@gmail.com <mailto:sam414255p...@gmail.com>>
> wrote:
>
>> Hi Chesnay
>>
>> If REST API (i.e. the web server) is mandatory for submitting
jobs then
>> why is there an option to set rest.port to -1? I think it should be
>> mandatory to set some valid port for rest.port and make sure
flink job
>> manager does not come up if valid port is not set for
rest.port? Or else
>> there must be some way to submit jobs even if REST API (i.e.
the web
>> server) is not instantiated.
>>
>> If jobmanger.rpc.address is not required for flink client then
why is it
>> still looking for that property in flink-conf.yaml? Isn't it
not a bug?
>> Because if we comment out the jobmanger.rpc.address and
jobmanger.rpc.port
>> then flink client will not be able to submit the job.
>>
>>
>> On Tue, Jun 19, 2018 at 5:49 PM, Chesnay Schepler
<ches...@apache.org <mailto:ches...@apache.org>>
>> wrote:
>>
>>> In 1.5 we reworked the job-submission to go through the REST
API instead
>>> of akka.
>>>
>>> I believe the jobmanager rpc port shouldn't be necessary
anymore, the rpc
>>> address is still *required *due to some technical
implementations; it
>>> may be that you can set this to some arbitrary value however.
>>>
>>> As a result the REST API (i.e. the web server) must be running
in order
>>> to submit jobs.
>>>
>>>
>>> On 19.06.2018 14:12, Sampath Bhat wrote:
>>>
>>> Hello
>>>
>>> I'm using Flink 1.5.0 version and Flink CLI to submit jar to flink
>>> cluster.
>>>
>>> In flink 1.4.2 only job manager rpc address and job manager
rpc port were
>>> sufficient for flink client to connect to job manager and
submit the job.
>>>
>>> But in flink 1.5.0 the flink client additionally requires the
>>> rest.address and rest.port for submitting the job to job
manager. What is
>>> the advantage of this new method over the 1.4.2 method of
submitting job?
>>>
>>> Moreover if we make rest.port = -1 the web server will not be
>>> instantiated then how should we submit the job?
>>>
>>> Regards
>>> Sampath
>>>
>>>
>>>
