[
https://issues.apache.org/jira/browse/SPARK-20871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16023306#comment-16023306
]
Glen Takahashi edited comment on SPARK-20871 at 5/24/17 5:46 PM:
-----------------------------------------------------------------
Just attached an example heapdump of Janino logging adding gigabytes worth of
strings to the heap
was (Author: [email protected]):
An example of Janino logging adding gigabytes worth of strings to the heap
> Only log Janino code in debug mode
> ----------------------------------
>
> Key: SPARK-20871
> URL: https://issues.apache.org/jira/browse/SPARK-20871
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 2.1.1
> Reporter: Glen Takahashi
> Attachments: 6a57e344-3fcf-11e7-85cc-52a06df2a489.png
>
>
> Currently if Janino code compilation fails, it will log the entirety of the
> code in the executors. Because the generated code can often be very large,
> the logging can cause heap pressure on the driver and cause it to fall over.
> I propose removing the "$formatted" from here:
> https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala#L964
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
