GitHub user smola opened a pull request:
https://github.com/apache/spark/pull/3645
[SPARK-4799] Use IP address instead of local hostname in ConnectionManager
See https://issues.apache.org/jira/browse/SPARK-4799
Spark fails when a node hostname is not resolvable by other nodes.
See an example trace:
```
14/12/09 17:02:41 ERROR SendingConnection: Error connecting to
27e434cf36ac:35093
java.nio.channels.UnresolvedAddressException
at sun.nio.ch.Net.checkAddress(Net.java:127)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:644)
at
org.apache.spark.network.SendingConnection.connect(Connection.scala:299)
at
org.apache.spark.network.ConnectionManager.run(ConnectionManager.scala:278)
at
org.apache.spark.network.ConnectionManager$$anon$4.run(ConnectionManager.scala:139)
```
The relevant code is here:
https://github.com/apache/spark/blob/bcb5cdad614d4fce43725dfec3ce88172d2f8c11/core/src/main/scala/org/apache/spark/network/nio/ConnectionManager.scala#L170
```
val id = new ConnectionManagerId(Utils.localHostName,
serverChannel.socket.getLocalPort)
```
This piece of code should use the host IP with Utils.localIpAddress or a
method that acknowleges user settings (e.g. `SPARK_LOCAL_IP`). Since I cannot
think about a use case for using hostname here, I'm creating a PR with the
former solution, but if you think the later is better, I'm willing to create a
new PR with a more elaborate fix.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/smola/spark SPARK-4799
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/3645.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3645
----
commit 45d0356241b852c575fb3b47a20c072c6ef3d7bf
Author: Santiago M. Mola <[email protected]>
Date: 2014-11-25T15:02:14Z
[SPARK-4799] Use IP address instead of local hostname in ConnectionManager.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
