Thank you Jacek. In case of YARN, I see that hadoop jars are present in system classpath for Driver. Will it be the same for all executors?
On Tue, Jun 7, 2016 at 11:22 AM, Jacek Laskowski <ja...@japila.pl> wrote: > Ouch, I made a mistake :( Sorry. > > You've asked about spark **history** server. It's pretty much the same. > > HistoryServer is a web interface for completed and running (aka > incomplete) Spark applications. It uses EventLoggingListener to > collect events as JSON using org.apache.spark.util.JsonProtocol > object. > > Pozdrawiam, > Jacek Laskowski > ---- > https://medium.com/@jaceklaskowski/ > Mastering Apache Spark http://bit.ly/mastering-apache-spark > Follow me at https://twitter.com/jaceklaskowski > > > On Tue, Jun 7, 2016 at 8:18 PM, Jacek Laskowski <ja...@japila.pl> wrote: > > Hi, > > > > It is the driver - see the port. Is this 4040 or similar? It's started > > when SparkContext starts and is controlled by spark.ui.enabled. > > > > spark.ui.enabled (default: true) = controls whether the web UI is > > started or not. > > > > It's through JobProgressListener which is the SparkListener for web UI > > that the console knows what happens under the covers (and can > > calculate the stats). > > > > BTW, spark.ui.port (default: 4040) controls the port Web UI binds to. > > > > Pozdrawiam, > > Jacek Laskowski > > ---- > > https://medium.com/@jaceklaskowski/ > > Mastering Apache Spark http://bit.ly/mastering-apache-spark > > Follow me at https://twitter.com/jaceklaskowski > > > > > > On Tue, Jun 7, 2016 at 8:11 PM, satish saley <satishsale...@gmail.com> > wrote: > >> Hi, > >> In spark history server, we see environment tab. Is it show environment > of > >> Driver or Executor or both? > >> > >> Jobs > >> Stages > >> Storage > >> Environment > >> Executors > >> >
