Dear all, I am currently looking at modifying NettyRpcEnv for this PR: https://github.com/apache/spark/pull/9608 The functionality which I’m trying to achieve is the following: if there is a configuration property spark.driver.advertisedPort, make executors reply to advertisedPort instead of spark.driver.port. This would enable NettyRpcEnv work correctly on Mesos with Docker bridge networking (something what is working spot on for AkkaRcpEnv).
I’ve spent some time looking at the new NettyRpcEnv and I think I know what is happening but seeking for confirmation. The master RpcEndpointAddress appears to be shipped to the executor as part of a serialized message, when the executors are requested, inside of the NettyRpcEndpointRef. In order to make my PR work, I need to change the RcpEndpointAddress shipped to the executors on the master. I think the best candidate is: https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/rpc/netty/NettyRpcEnv.scala#L125 Am I correct that this value becomes the RcpEndpointAddress? If so, my naive implementation would be like this: if (server != null) RpcAddress(host, conf.getInt(“spark.driver.advertisedPort”, server.getPort())) else null however, I am not sure what impact such change would have, about to test my changes as soon as the code from master branch builds. If that is not the correct place to make such change, what would be the best place to investigate? Is there an overview of NettyRcpEnv available anywhere? Kind regards, Radek Gruchalski ra...@gruchalski.com (mailto:ra...@gruchalski.com) (mailto:ra...@gruchalski.com) de.linkedin.com/in/radgruchalski/ (http://de.linkedin.com/in/radgruchalski/) Confidentiality: This communication is intended for the above-named person and may be confidential and/or legally privileged. If it has come to you in error you must take no action based on it, nor must you copy or show it to anyone; please delete/destroy and inform the sender immediately.
