Hi Nicholas,
I think you are right about the issue relating to Spark-11126, I'm seeing
it as well.
Did you find any workaround? Looking at the pull request for the fix it
doesn't look possible.
Best regards,
Patrick
On 15 October 2015 at 19:40, Nicholas Pritchard <
nicholas.pritch...@falkonry.
Thanks for your help, most likely this is the memory leak you are fixing in
https://issues.apache.org/jira/browse/SPARK-11126.
-Nick
On Mon, Oct 12, 2015 at 9:00 PM, Shixiong Zhu wrote:
> In addition, you cannot turn off JobListener and SQLListener now...
>
> Best Regards,
> Shixiong Zhu
>
> 201
In addition, you cannot turn off JobListener and SQLListener now...
Best Regards,
Shixiong Zhu
2015-10-13 11:59 GMT+08:00 Shixiong Zhu :
> Is your query very complicated? Could you provide the output of `explain`
> your query that consumes an excessive amount of memory? If this is a small
> quer
Is your query very complicated? Could you provide the output of `explain`
your query that consumes an excessive amount of memory? If this is a small
query, there may be a bug that leaks memory in SQLListener.
Best Regards,
Shixiong Zhu
2015-10-13 11:44 GMT+08:00 Nicholas Pritchard <
nicholas.prit
As an update, I did try disabling the ui with "spark.ui.enabled=false", but
the JobListener and SQLListener still consume a lot of memory, leading to
OOM error. Has anyone encountered this before? Is the only solution just to
increase the driver heap size?
Thanks,
Nick
On Mon, Oct 12, 2015 at 8:4
I set those configurations by passing to spark-submit script:
"bin/spark-submit --conf spark.ui.retainedJobs=20 ...". I have verified
that these configurations are being passed correctly because they are
listed in the environments tab and also by counting the number of
job/stages that are listed. T
Could you show how did you set the configurations? You need to set these
configurations before creating SparkContext and SQLContext.
Moreover, the history sever doesn't support SQL UI. So
"spark.eventLog.enabled=true" doesn't work now.
Best Regards,
Shixiong Zhu
2015-10-13 2:01 GMT+08:00 pnpritc
