[
https://issues.apache.org/jira/browse/SPARK-19720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16105427#comment-16105427
]
Mark Grover commented on SPARK-19720:
-------------------------------------
I wasn't planning on. One could argue the case that this could be backported to
branch-2.1 given that it's a rather simple change. However, 2.2 brought in some
changes that were long overdue - dropping support for Java 7, Hadoop 2.5 and
even if we got this change backported, you won't be able to make use of
goodness down the road you didn't upgrade to Hadoop 2.6, Java 8, etc. So, my
recommendation here would be to brave the new world of hadoop 2.6.
> Redact sensitive information from SparkSubmit console output
> ------------------------------------------------------------
>
> Key: SPARK-19720
> URL: https://issues.apache.org/jira/browse/SPARK-19720
> Project: Spark
> Issue Type: Bug
> Components: Spark Submit
> Affects Versions: 2.2.0
> Reporter: Mark Grover
> Assignee: Mark Grover
> Fix For: 2.2.0
>
>
> SPARK-18535 took care of redacting sensitive information from Spark event
> logs and UI. However, it intentionally didn't bother redacting the same
> sensitive information from SparkSubmit's console output because it was on the
> client's machine, which already had the sensitive information on disk (in
> spark-defaults.conf) or on terminal (spark-submit command line).
> However, it seems now that it's better to redact information from
> SparkSubmit's console output as well because orchestration software like
> Oozie usually expose SparkSubmit's console output via a UI. To make matters
> worse, Oozie, in particular, always sets the {{--verbose}} flag on
> SparkSubmit invocation, making the sensitive information readily available in
> its UI (see
> [code|https://github.com/apache/oozie/blob/master/sharelib/spark/src/main/java/org/apache/oozie/action/hadoop/SparkMain.java#L248]
> here).
> This is a JIRA for tracking redaction of sensitive information from
> SparkSubmit's console output.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
