w.r.t. getting application Id, please take a look at the following in SparkContext :
/** * A unique identifier for the Spark application. * Its format depends on the scheduler implementation. * (i.e. * in case of local spark app something like 'local-1433865536131' * in case of YARN something like 'application_1433865536131_34483' * ) */ def applicationId: String = _applicationId On Mon, Dec 14, 2015 at 2:33 PM, Jonathan Kelly <jonathaka...@gmail.com> wrote: > Are you running Spark on YARN? If so, you can get to the Spark UI via the > YARN ResourceManager. Each running Spark application will have a link on > the YARN ResourceManager labeled "ApplicationMaster". If you click that, it > will take you to the Spark UI, even if it is running on a slave node in the > case of yarn-cluster mode. It does this by proxying the Spark UI through > the YARN Proxy Server on the master node. > > For completed applications, the link will be labeled "History" and will > take you to the Spark History Server (provided you have > set spark.yarn.historyServer.address in spark-defaults.conf). > > As for getting the URL programmatically, the URL using the YARN > ProxyServer is easy to determine. It's just http://<master node > address>:<YARN ProxyServer port>/proxy/<YARN application ID>. (e.g., > http://ip-10-150-65-11.ec2.internal:20888/proxy/application_1450128858020_0001/) > Then again, I'm not sure how easy it is to get the YARN application ID for > a Spark application without parsing the spark-submit logs. Or at least I > think I remember some other thread where that was mentioned. > > ~ Jonathan > > On Mon, Dec 14, 2015 at 1:57 PM, Ashish Nigam <ashnigamt...@gmail.com> > wrote: > >> Hi, >> I run spark streaming job in cluster mode. This means that driver can run >> in any data node. And Spark UI can run in any dynamic port. >> At present, I know about the port by looking at container logs that look >> something like this - >> >> server.AbstractConnector: Started SelectChannelConnector@0.0.0.0:50571 >> INFO util.Utils: Successfully started service 'SparkUI' on port 50571. >> INFO ui.SparkUI: Started SparkUI at http://xxx:50571 >> >> >> Is there any way to know about the UI port automatically using some API? >> >> Thanks >> Ashish >> > >
