Hi,
I'm not surprised to see Hadoop jars on the driver (yet I couldn't
explain exactly why they need to be there). I can't find a way now to
display the classpath for executors.
Pozdrawiam,
Jacek Laskowski
https://medium.com/@jaceklaskowski/
Mastering Apache Spark http://bit.ly/mastering-apa
Thank you Jacek.
In case of YARN, I see that hadoop jars are present in system classpath for
Driver. Will it be the same for all executors?
On Tue, Jun 7, 2016 at 11:22 AM, Jacek Laskowski wrote:
> Ouch, I made a mistake :( Sorry.
>
> You've asked about spark **history** server. It's pretty much
Ouch, I made a mistake :( Sorry.
You've asked about spark **history** server. It's pretty much the same.
HistoryServer is a web interface for completed and running (aka
incomplete) Spark applications. It uses EventLoggingListener to
collect events as JSON using org.apache.spark.util.JsonProtocol
Hi,
It is the driver - see the port. Is this 4040 or similar? It's started
when SparkContext starts and is controlled by spark.ui.enabled.
spark.ui.enabled (default: true) = controls whether the web UI is
started or not.
It's through JobProgressListener which is the SparkListener for web UI
that
