Thanks for the reply. I am now trying to configure yarn.web-proxy.address according to https://issues.apache.org/jira/browse/SPARK-5837, but cannot start the standalone web proxy server.
I am using CDH 5.0.1 and below is the error log: sbin/yarn-daemon.sh: line 44: /opt/cloudera/parcels/CDH/lib/hadoop-yarn/sbin/../libexec/yarn-config.sh: No such file or directory mkdir: cannot create directory `/logs': Permission denied chown: cannot access `/logs': No such file or directory starting proxyserver, logging to /logs/yarn-xujun-proxyserver-nfjd-hadoop02-node26.jpushoa.com.out sbin/yarn-daemon.sh: line 124: /logs/yarn-xujun-proxyserver-nfjd-hadoop02-node26.jpushoa.com.out: No such file or directory head: cannot open `/logs/yarn-xujun-proxyserver-nfjd-hadoop02-node26.jpushoa.com.out' for reading: No such file or directory sbin/yarn-daemon.sh: line 129: /logs/yarn-xujun-proxyserver-nfjd-hadoop02-node26.jpushoa.com.out: No such file or directory sbin/yarn-daemon.sh: line 130: /logs/yarn-xujun-proxyserver-nfjd-hadoop02-node26.jpushoa.com.out: No such file or directory I tried running `sudo sbin/yarn-daemon.sh start proxyserver` to give permission to create the directory /logs, but then it reported /bin/yarn not found. Seems the installation and configuration of CDH distribution has something different with the Apache one. 2016-03-04 18:27 GMT+08:00 Steve Loughran <ste...@hortonworks.com>: > > On 3 Mar 2016, at 09:17, Shady Xu <shad...@gmail.com> wrote: > > Hi all, > > I am running Spark in yarn-client mode, but every time I access the web > ui, the browser redirect me to one of the worker nodes and shows nothing. > The url looks like > http://hadoop-node31.company.com:8088/proxy/application_1453797301246_120264 > . > > > > that should be acting as a proxy for the web UI; the way YARN manages > security and auth through the proxy; the spark Web UI runs a filter which > redirects all queries that don't come from the proxy server to that proxy > server > > It should really be picking up the address of that proxy from the hadoop > cluster configuration option yarn.web-proxy.address , falling back to the > value of yarn.resourcemanager.webapp.address if it can't find that value > (it's actually a bit more complex than that, but you'd have to look at > Hadoop's org.apache.hadoop.yarn.webapp.util.WebAppUtils for the full details > > > > I googled a lot and found some possible bugs of YARN which may cause this > problem. But still want to know whether there is a way to show the web ui > because it is very inconvenient to debug. > > > > It should be correctly linked through the yarn RM web UI, the one where > you get a list of all running applications. If not: what about any other > apps running in the cluster? Do they link up properly? >
