Jerry, Thanks for your comment.
On Mon, Sep 4, 2017 at 10:43 AM, Saisai Shao <sai.sai.s...@gmail.com> wrote: > I think spark.yarn.am.port is not used any more, so you don't need to > consider this. > > If you're running Spark on YARN, I think some YARN RM port to submit > applications should also be reachable via firewall, as well as HDFS port to > upload resources. > > Also in the Spark side, executors will be connected to driver via > spark.driver.port, maybe you should also set a fixed port number for this > and add to white list of firewall. > > Thanks > Jerry > > > On Mon, Sep 4, 2017 at 8:50 AM, Satoshi Yamada < > satoshi.yamada....@gmail.com> wrote: > >> Hi, >> >> In case we run Spark on Yarn in client mode, we have firewall for Hadoop >> cluster, >> and the client node is outside firewall, I think I have to open some ports >> that Application Master uses. >> >> >> I think the ports is specified by "spark.yarn.am.port" as document says. >> https://spark.apache.org/docs/latest/running-on-yarn.html >> >> But, according to the source code, spark.yarn.am.port is deprecated since >> 2.0. >> https://github.com/apache/spark/commit/829cd7b8b70e65a91aa66e6d626bd45f18e0ad97 >> >> Does this mean we do not need to open particular ports of firewall for >> >> Spark on Yarn? >> >> >> Thanks, >> >> >
