Re: Spark UI crashes on Large Workloads

Hi Riccardo, Thanks for your suggestions. The thing is that my Spark UI is the one thing that is crashing - and not the app. In fact the app does end up completing successfully. That's why I'm a bit confused by this issue? I'll still try out some of your suggestions. Thanks and Regards, Saatvik Sh

Re: Spark UI crashes on Large Workloads

The reason you get connection refused when connecting to the application UI (port 4040) is because you app gets stopped thus the application UI stops as well. To inspect your executors logs after the fact you might find useful the Spark History server

Re: Spark UI crashes on Large Workloads

Hi Riccardo, Yes, Thanks for suggesting I do that. [Stage 1:==> (12750 + 40) / 15000]17/07/18 13:22:28 ERROR org.apache.spark.scheduler.LiveListenerBus: Dropping SparkListenerEvent because no remaining room in event queue. This likely means one of the

Re: Spark UI crashes on Large Workloads

Hi, can you share more details. do you have any exceptions from the driver? or executors? best, On Jul 18, 2017 02:49, "saatvikshah1994" wrote: > Hi, > > I have a pyspark App which when provided a huge amount of data as input > throws the error explained here sometimes: > https://stackoverflow