@Andrew
Yes, the link point to the same redirected
http://localhost/proxy/application_1404443455764_0010/
I suspect something todo with the cluster setup. I will let you know
once I found something.
Chester
On Mon, Jul 7, 2014 at 1:07 PM, Andrew Or wrote:
> @Yan, the UI shoul
Thank you, Andrew. That makes sense for me now. I was confused by "In
yarn-cluster mode, the Spark driver runs inside an application master
process which is managed by YARN on the cluster" in
http://spark.apache.org/docs/latest/running-on-yarn.html . After
you explanation, it's clear now. Thank you
@Yan, the UI should still work. As long as you look into the container that
launches the driver, you will find the SparkUI address and port. Note that
in yarn-cluster mode the Spark driver doesn't actually run in the
Application Manager; just like the executors, it runs in a container that
is launc
As Andrew explained, the port is random rather than 4040, as the the spark
driver is started in Application Master and the port is random selected.
But I have the similar UI issue. I am running Yarn Cluster mode against my
local CDH5 cluster.
The log states
"14/07/07 11:59:29 INFO ui.SparkUI: St
Hi Andrew,
Thanks for the quick reply. It works with the yarn-client mode.
One question about the yarn-cluster mode: actually I was checking the AM
for the log, since the spark driver is running in the AM, the UI should
also work, right? But that is not true in my case.
Best,
Fang, Yan
yanfang.
I will assume that you are running in yarn-cluster mode. Because the driver
is launched in one of the containers, it doesn't make sense to expose port
4040 for the node that contains the container. (Imagine if multiple driver
containers are launched on the same node. This will cause a port
collisio
Hi guys,
Not sure if you have similar issues. Did not find relevant tickets in
JIRA. When I deploy the Spark Streaming to YARN, I have following two
issues:
1. The UI port is random. It is not default 4040. I have to look at the
container's log to check the UI port. Is this suppose to be this wa
