[
https://issues.apache.org/jira/browse/SPARK-20871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16090125#comment-16090125
]
PJ Fanning commented on SPARK-20871:
------------------------------------
[~srowen] One of the main reasons for the Janino compile to fail is if the
generated code is too large (> 64k). In this case, printing the full code is
not as useful.
The code is still logged in full at debug level, so users can always
reconfigure their logging to get the full logging.
In my team's use case, the fallback to not using codegen works ok and the
codegen is still so fast that it suits us best to allow the codegen and have
the automatic fallback in the edge case of the codegen failing. We have a SaaS
application that uses Spark and we don't really want our user's code to appear
in our logs by default.
> Only log Janino code in debug mode
> ----------------------------------
>
> Key: SPARK-20871
> URL: https://issues.apache.org/jira/browse/SPARK-20871
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 2.1.1
> Reporter: Glen Takahashi
> Priority: Trivial
> Attachments: 6a57e344-3fcf-11e7-85cc-52a06df2a489.png
>
>
> Currently if Janino code compilation fails, it will log the entirety of the
> code in the executors. Because the generated code can often be very large,
> the logging can cause heap pressure on the driver and cause it to fall over.
> I propose removing the "$formatted" from here:
> https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala#L964
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
