Re: Spark UI crashes on Large Workloads

2017-07-18 Thread Saatvik Shah
02:49, "saatvikshah1994" >>> wrote: >>> >>>> Hi, >>>> >>>> I have a pyspark App which when provided a huge amount of data as input >>>> throws the error explained here sometimes: >>>> https://stackoverflow.

Re: Spark UI crashes on Large Workloads

2017-07-18 Thread Riccardo Ferrari
tly >>> peculiar thing I am doing in this app is using a custom PySpark ML >>> Transformer(Modified from >>> https://stackoverflow.com/questions/32331848/create-a-custom >>> -transformer-in-pyspark-ml). >>> Could this be the issue? How can I d

Re: Spark UI crashes on Large Workloads

2017-07-18 Thread Saatvik Shah
a custom PySpark ML >> Transformer(Modified from >> https://stackoverflow.com/questions/32331848/create-a-custom >> -transformer-in-pyspark-ml). >> Could this be the issue? How can I debug why this is happening? >> >> >> >> -- >> View this mes

Re: Spark UI crashes on Large Workloads

2017-07-17 Thread Riccardo Ferrari
gt; Transformer(Modified from > https://stackoverflow.com/questions/32331848/create-a- > custom-transformer-in-pyspark-ml). > Could this be the issue? How can I debug why this is happening? > > > > -- > View this message in context: http://apache-spark-user-list. > 1001560.n3.nabble.com/

Spark UI crashes on Large Workloads

2017-07-17 Thread saatvikshah1994
: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-UI-crashes-on-Large-Workloads-tp28873.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe e-mail: user-unsubscr