Github user WangTaoTheTonic commented on the pull request:
https://github.com/apache/spark/pull/3112#issuecomment-62090652
Here is the scenario: When a client submit an application on a node which
is not in yarn cluster. The ApplicationMaster may not recognize the client's
hostname like this:
`14/11/07 11:11:02 INFO ApplicationMaster: Waiting for Spark driver to be
reachable.
14/11/07 11:11:02 ERROR ApplicationMaster: Failed to connect to driver at
datasight1:10084, retrying ...
14/11/07 11:11:02 ERROR ApplicationMaster: Failed to connect to driver at
datasight1:10084, retrying ...
14/11/07 11:11:02 ERROR ApplicationMaster: Failed to connect to driver at
datasight1:10084, retrying ...`
I think it happens when Application Master could not resolve client's
hostname(because the client's hostname-ip pair was not added into /etc/hosts or
DNS server).
If we set `spark.driver.host` AM will use the config value(for instance,
client's ip address) instead of the default value(Utils.localHostName()), then
the error will not occur.
That is to say, actually in yarn-client mode this configuration is useful
sometime.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
